Tools → Schema Manager → Schemas
Schemas list view
By default, the Schema items appear in alphabetical, ascending order. The list shows 40 items at a time. To sort the items by the column values in either ascending or descending order, select a specific column header. For any schema row, select any column other than the Status and Last Load links, to open the item in the Schema Designer. The Schemas tab shows the size of the data on disk. The Refresh (circle with arrow icon) is available to refresh the list view and recalculate the data size.
When you select one or more physical schema, the Search bar shows the More Options button (⋮ vertical ellipsis). The available options are:
- Export: Exports the selected physical schemas to an Incorta package.
- Add Load Plan: Opens the New Load Plan dialog to create a load plan for the selected physical schemas.
- Delete: Deletes all selected physical schemas.
- You can delete only items that you have Edit access rights to.
For Schemas, the List view shows the following properties:
Property | Control | Description |
---|---|---|
Select All | checkbox | Header control to add or remove all list view page items to a group for deletion, export, or scheduled loading |
Select Item | checkbox | Select to add or remove an individual item to a group of zero or more items for deletion, export, or scheduled loading |
Name | text | The name of the physical schema |
Load Status | link | The status of the last or current load job. Hover over it to display the time spent for each phase in the load job. Select it to go to the job details in the Load Job Details Viewer. The available statuses are: ● Succeeded ● Finished With Errors ● Failed ● Interrupted ● In Progress (Schemas with a job that is running or in planning) ● In Queue |
Last Successful Load | timestamp | The date and time of the last successful load job for the physical schema, if any |
Last Load | link | A letter that represents the type of the last or current load job and its start date and time, if any, regardless of the job status. The letter is one of the following: ● F for Full load ● I for Incremental load ● S for Staging Select the timestamp link, if any, to go to the job details. |
Next Load | timestamp | The type and the date and time for the next scheduled load job, if any. If the schema is not part of any load plan, it shows Not Scheduled. |
Model Status | link | The status of the last or current schema update job (model update job). Hover over it to display the time spent for each phase in the update job. Select it to go to the Model Update Viewer. For newly added physical schemas (whether created or imported without overwriting existing ones), no model status is available. If one or more of the physical schema objects have changes that require loading data, there will be a triangle with an exclamation mark next to the status till the required load occurs, whether manually or through a scheduled job. If the current update job is still in queue because one or more load or update jobs are blocking it, the Model Status (In Queue) will be a link that opens a pop-up that shows a list of these blocking jobs. |
Data Size | text | The size of the physical schema data that is loaded into the engine memory |
Owner | text | The display name of the user creator |
Modified By | text | The display name of the user that last modified the physical schema |
More Options → Edit Description | menu link | Opens the Edit Description dialog |
More Options → Performance Optimization | menu link | Opens the Optimize Table Performance dialog |
More Options → Change Load Order | menu link | Opens the Choose Load Order dialog |
More Options → Add Load Plan | menu link | Opens the New Load Plan dialog |
More Options → Share | menu link | Opens the Share dialog |
More Options → Export | menu link | Opens the Export Schema(s) dialog |
More Options → Delete | menu link | Opens the confirmation dialog for the physical schema deletion |
Before the 2024.1.0 release, you could use the Schema Manager to create a scheduled job to send email notifications with the status of loading one or more schemas. Starting with the 2024.1.x releases, load plan notifications have replaced schema load notifications, allowing for creating email notifications at the load plan level rather than the schema level. You can use the Scheduler to create load plan notifications.
For physical schemas with dependencies, a physical schema load or update job can block the load and update jobs of dependent physical schemas. Blocked update jobs stay in the In Queue state until all blocking jobs are completed while the blocked load jobs stay in the Load state. You can view a list of jobs that block update jobs. For more details, see View blocking jobs.
Aborted
is the status of a load job that is automatically forced to stop after exceeding the configured interruption timeout. Incorta automatically aborts any load job that is interrupted in the extraction phase and exceeds the configured interruption time interval.
Schema Manager actions for physical schemas
With the Schema Manager, you can perform the following actions for physical schemas:
- Create a physical schema with the Schema Wizard
- Create a physical schema with the Schema Designer
- Import a physical schema
- View a list of physical schemas
- Search for a physical schema
- Filter physical schemas by the load status
- Filter physical schemas by the model update status
- Edit a physical schema
- View physical schema details
- View the load job details for a physical schema
- View the physical schema update jobs
- View blocking jobs
- Add or edit a physical schema description
- Optimize performance for physical schema objects
- Change the load order of physical schema objects
- Create a load plan
- Share a physical schema
- Export a physical schema
- Export one or more physical schemas
- Delete a physical schema
- Delete one or more physical schemas
Schema Manager access rights matrix
As the Super User or a user that belongs to a group with the Schema Manager or the SuperRole role, you can access the Schema Manager. You can import physical schemas and create them, whether with the Schema Wizard or the Schema Designer. However, to be able to view, search for, or manage items on this page, you need to have the suitable access rights to them.
The following table shows the required access rights for each action.
Action/Access Rights | Owner/Edit | Share | View |
---|---|---|---|
Schema → View on list | ✔ | ✔ | ✔ |
Schema → Search for or filter | ✔ | ✔ | ✔ |
Schema → View details | ✔ | ✔ | ✔ |
Schema → View Description | ✔ | ✔ | ✔ |
Schema → Add/Edit Description | ✔ | ||
Schema → Edit details | ✔ | ||
Schema → Export | ✔ | ✔ | ✔ |
Schema → Optimize table performance | ✔ | View only | View only |
Schema → Manage load order | ✔ | View only | View only |
Schema → Create load plan | ✔ | ||
Schema → Share | ✔ | ✔ | View only |
Schema → Delete | ✔ |
- When the Cluster Management Console (CMC) admin turns the Enable Super User Mode option on, the Super User and users with the SuperRole can manage all physical schemas regardless of the access rights.
- Only the Super User tenant administrator can view the list of all items on the Schema Manager regardless of the access rights to them and the Enable Super User mode option.
Create a physical schema with the Schema Wizard
A physical schema name…
- Must be unique.
- Must be between 1 and 250 characters in length.
- Must begin with an alpha character, lower or upper case.
- After the first alpha character, can contain zero or more alphanumeric characters in lower, upper, or mixed case.
- After the first alpha character, can contain zero or more underscore
(_)
or Dollar sign($)
characters. - Besides underscore
(_)
and Dollar sign($)
, cannot contain special characters, symbols, or spaces. - Is not case-sensitive. (Starting with 2024.1.x)
Here are the steps to create a physical schema with the Schema Wizard:
- On the Navigation bar, select Schema.
- On the Action bar, select + New → Schema Wizard.
- In (1) Choose a Source, specify the following:
- In Enter a name, enter the physical schema name.
- In Select a Datasource, select the external data source.
- Optionally, create a description.
- In the Schema Wizard footer, select Next.
- In (2) Manage Tables, in the Data panel, select the name of the Data Source, and then select the tables, files, or worksheets you want or select the Select All checkbox to select all tables, files, or worksheets depending upon the data source you selected. For each selected table, you can change the table name in Incorta in releases starting 2024.7.x.
- In the Schema Wizard footer, select Next.
- In (3) Finalize, in the Schema Wizard footer, select Create Schema.
Create a physical schema with the Schema Designer
Before creating a physical schema with the Schema Designer, review References → Connectors → for specific table property definitions.
Here are the steps to create a physical schema using the Schema Designer:
- On the Navigation bar, select Schema.
- On the Action bar, select + New → Create Schema.
- In Name, specify the physical schema name, and select Save.
- In Start adding tables to your physical schema, select one of the following:
- Schema Wizard: to use the Schema Wizard to select the tables, files, or worksheets you want to add
- Incorta Analyzer: to use the Analyzer to create a table or aggregated table insight as a physical schema Incorta Analyzer table
- Incorta SQL: to use Incorta SQL Engine to create an Incorta SQL table by querying objects in the same or other physical schemas.
- SQL Table: to create a SQL table from different data source types, such as MySQL, Oracle, IBM DB2, Cassandra (Simba), NetSuite Suite Analytics, Presto, SAP Hana, and BigQuery
- File System: to create a table from a local data file
- Cassandra: to create a table from an Apache Cassandra data source
- Cisco Meraki: to create a table from a Cisco Meraki data source
- Data Lake: to create a table from a Data Lake data source
- DynamoDB: to create a table from an AWS DynamoDB data source
- Essbase: to create a table from an Oracle Essbase data source
- Freshservice: to create a table from a Freshservice data source
- Google Sheets: to create a table from a Google Sheets data source
- Kafka: to create a table from an Apache Kafka data source
- NetSuite: to create a table from a NetSuite Web Service data source
- OneLogin: to create a table from a OneLogin data source
- Oracle Cloud Applications: to create a table from an Oracle Cloud Application data source
- SalesForce: to create a table from a Salesforce data source
- ServiceNow: to create a table from a ServiceNow data source
- SharePoint: to create a table from a SharePoint data source
- Splunk: to create a table from a Splunk data source
- Zuora: to create a table from a Zuora data source
- Materialized: to create a Materialized View
- Alias: to create an alias that references a selected physical schema object, in the same physical schema or other physical schemas
- In the Data Source dialog, specify the properties for the given object.
- Select Add.
- In the Table Editor, in the Table Summary section, enter the physical schema object name.
- To save your changes, on the Action bar, select Done.
Import a physical schema
You can import one or more physical schemas that exist in the same .zip
file. When you import a physical schema, if there is no physical schema that shares the same name with the imported one, a new physical schema is created from this imported one with no version. If you try to import a physical schema that shares the name with an existing one, the import fails. However, you can overwrite this existing physical schema. When you overwrite a physical schema that has saved versions (whether named or not), these versions are not deleted, and a new saved version is added as the current version.
You cannot import a physical schema with the Overwrite existing schema option enabled if another user is editing it in the draft mode nor during a schema update job.
Here are the steps to import a physical schema:
- On the Navigation bar, select Schema.
- On the Action bar, select + New → Import Schema.
- In the Import Schema dialog, optionally select the Overwrite existing schema checkbox to overwrite any existing physical schema that shares the same name with the imported one, if any.
- Do one of the following:
- Drag the
.zip
file that contains the physical schema(s) that you want to import to the Import Schema dialog. - Select Click or drag a file here to upload to select the
.zip
file, and then select Open.
- Drag the
- Wait until the physical schema(s) and scheduled load job(s), if any, are imported.
- In the Import Results dialog, review the status of the import operation.
- Select Close.
The following are the details in the Import Results dialog:
Property | Description |
---|---|
Name | The name of the physical schema or scheduled load job |
Type | The type of the imported item: Schema or Scheduler |
Status | The status of the import operation. It can be one of the following: ● Created ● Overwritten ● Failed If the import operation fails, the reason is displayed. If the physical schema import operation fails, related scheduled load jobs are not imported even if the same physical schema already exists. |
- Importing a schema with its load plans will overwrite the single-schema load plans only.
View a list of physical schemas
- On the Navigation bar, select Schema.
- On the Schemas tab, review the list view of physical schemas.
Search for a physical schema
- On the Navigation bar, select Schema.
- On the Schemas tab, on the Search bar, enter a search term.
- Review the list view of physical schemas.
To reset the search, on the Search bar, in the search box, select the remove button (x).
Filter physical schemas by the load status
- On the Navigation bar, select Schema.
- On the Schemas tab, in the list view header row, select the filter icon next to Load Status.
- Select one or more load statuses to filter the physical schemas list view.
The load status options are:
- Succeeded
- Finished With Errors
- Failed
- Interrupted
- In Progress (Running or in Planning)
- In Queue
- Select OK.
Reset the load status filter
- On the Navigation bar, select Schema.
- On the Schemas tab, in the list view header row, select the filter icon next to Load Status.
- Select Reset.
Filter physical schemas by the model update status
- On the Navigation bar, select Schema.
- On the Schemas tab, in the list view header row, select the filter icon next to Model Status.
- Select one or more model update statuses to filter the physical schemas list view. The model update status options are:
- Succeeded
- Finished With Errors
- Failed
- Interrupted
- In Progress
- In Queue
- Select OK.
Reset the model update status filter
- On the Navigation bar, select Schema.
- On the Schemas tab, in the list view header row, select the filter icon next to Model Status.
- Select Reset.
Edit a physical schema
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema that you own or have edit access rights to, select any part of the row except the Status and Last Load links.
- In the Schema Designer, you can do the following:
- view a list of the physical schema objects and joins
- add, edit, or remove physical schema objects
- add, edit, or remove joins between the physical schema objects
- load data for the physical schema
- load data for a selected physical schema object
- view or manage load jobs
- open the Table Editor for a selected physical schema object
- switch to the Diagram view
- open the Join Editor for a select join
- explore physical schema data
- add or edit a physical schema description
- optimize performance for the physical schema objects
- change the load order of the physical schema objects
- create a load plan
- share a physical schema
View physical schema details
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema, select any part of the row except the Status and Last Load links.
- In the Schema Designer, you can do the following:
- view load jobs
- view physical schema objects and joins
- view physical schema object details, such as the columns, formula columns, and load filters
- explore physical schema data
- view the physical schema diagram
- view or edit the share options for the physical schema depending upon the access rights
- view the physical schema description, objects load order, and performance optimization settings for the physical schema objects
For more information, refer to Tools → Schema Designer
View the load job details for a physical schema
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema, select the Load Status or Last Load link.
- In the Load Job Details Viewer, review the details of the load jobs for the physical schema.
You can track all load jobs and load plans related to a physical schema on the Schema Manager → Load Jobs.
View the physical schema update jobs
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema, select the Model Status link.
- In the Model Update Viewer, review the details of the physical schema update jobs.
View blocking jobs
When an update job or a load job is in the commit phase or the load and post-load phase, respectively, it blocks the update and load jobs on dependent physical schemas. A dependent physical schema is one that has a cross join relationship with another physical schema. The blocked update jobs stay in the In Queue state till all blocking jobs are completed while the load jobs stay in the Load state. As a schema manager user, you can view a list of jobs that block update jobs in the Schema Manager and the Model Update Viewer. This list shows the blocking physical schema, the type of the blocking job, load or update, and the current status of each blocking job. You can also access the details of the blocking job in the Load Job Details Viewer or the Model Update Viewer according to the type of the blocking job.
Here are the steps to view the list of blocking jobs of an update job in the Schema Manager:
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema that has a blocked update job, in Model Status, select the In Queue link. A pop-up dialog appears with a list of the blocking jobs.
- To view the details of a blocking job, in the pop-up dialog, select See Job Details.
Add or edit a physical schema description
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
- Select Edit Description.
- In the Edit Description dialog, enter or update the physical schema description. You can enter up to 500 alphanumeric characters, including spaces and special characters.
- Select Save.
Optimize performance for physical schema objects
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
- Select Performance Optimization.
- In the Optimize Table Performance dialog, in the objects list, for each object you want to optimize the performance, select the checkbox, and for each object that you do not want to optimize the performance, clear the checkbox.
- Select Save.
You must enable performance optimization for Incorta Analyzer tables and Incorta SQL tables to avoid load failure.
For objects that you enable for performance optimization, the Loader Service will load their files from Shared Storage (Staging) into the Engine memory. For unselected objects, the Loader Service will load data only into Shared Storage, which can slow the speed of creating insights; however, this will reduce the load time.
Change the load order of physical schema objects
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
- Select Change Load Order.
- In the Choose Load Order dialog, you can do the following to manage the
table groups before changing their load order:
- For each table or Materialized View that you want to add to a new load group, select the checkbox, and then select Add to New Group.
- For each table or Materialized View that you want to add to an existing load group, select the checkbox, and then select Add to Existing Group.
- To reset table grouping and order, select Revert to Default.
- To set the order of a table group, select the up arrow (to the right of the group name) to move the group upwards, or select the down arrow (to the right of the group name) to move the group downwards. Repeat this action until you order the table groups as required.
- Select Save.
Share a physical schema
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
- Select Share.
- In the Share dialog, you can do the following:
- remove an existing user or group
- modify the existing access rights for a user or group
- add a new user or group with the required access rights
Export a physical schema
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
- Select Export.
- In the Export Schema(s) dialog, optionally select the Include load plans checkbox to include scheduled load jobs or load plans related to the selected physical schema in the export operation.
- Select Export. The exported physical schema
.zip
file is downloaded to the Downloads folder.
- When you export a physical schema, you export the current saved version only. No drafts or old saved versions are exported. To export a previous saved version, use the Schema Designer, for more details, refer to Export a saved version.
- Exporting a schema with its load plans will include only single-schema load plans that include this schema.
Export one or more physical schemas
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for each physical schema that you want to export, select the checkbox.
- On the Search bar, next to Selected, select More Options (⋮ vertical ellipsis).
- Select Export.
- In the Export Schema(s) dialog, optionally select the Include load plans checkbox to include scheduled load jobs or load plans related to the selected physical schema(s) in the export operation.
- Select Export. The exported physical schema(s)
.zip
file is downloaded to the Downloads folder.
Delete a physical schema
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
- Select Delete.
- In the dialog, select Delete.
- When you delete a schema, the history of scheduled load plans of this schema will be maintained in the Schema Manager > Load Jobs page.
- You cannot delete a schema if another user is editing it in the draft mode.
- You cannot delete a schema if it is part of a running load job, whether scheduled or manual.
- When you delete a schema that exists in a single-schema load plan, the load plan will be automatically deleted along with its scheduled runs.
- When you delete a schema that exists in a multi-schema load plan:
- If the schema exists in a load group with other schemas, the schema will be removed from the group.
- If the schema exists alone in a load group, the group will be removed from the load plan.
Delete one or more physical schemas
- On the Navigation bar, select Schema.
- In the list view of physical schemas, for each physical schema that you want to delete, select the checkbox.
- On the Search bar, next to Selected, select More Options (⋮ vertical ellipsis).
- Select Delete.
- In the dialog, select Delete.