Tools → Schema Manager → Schemas

Schemas list view

By default, the Schema items appear in alphabetical, ascending order. The list shows 40 items at a time. To sort the items by the column values in either ascending or descending order, select a specific column header. For any schema row, select any column other than the Status and Last Load links, to open the item in the Schema Designer. The Schemas tab shows the size of the data on disk. The Refresh (circle with arrow icon) is available to refresh the list view and recalculate the data size.

When you select one or more physical schema, the Search bar shows the More Options button (⋮ vertical ellipsis). The available options are:

  • Export: Exports the selected physical schemas to an Incorta package.
  • Add Load Plan: Opens the New Load Plan dialog to create a load plan for the selected physical schemas.
  • Delete: Deletes all selected physical schemas.
Note
  • You can delete only items that you have Edit access rights to.

For Schemas, the List view shows the following properties:

PropertyControlDescription
Select AllcheckboxHeader control to add or remove all list view page items to a group for deletion, export, or scheduled loading
Select ItemcheckboxSelect to add or remove an individual item to a group of zero or more items for deletion, export, or scheduled loading
NametextThe name of the physical schema
Load StatuslinkThe status of the last or current load job. Hover over it to display the time spent for each phase in the load job. Select it to go to the job details in the Load Job Details Viewer.
The available statuses are:
  ●  Succeeded
  ●  Finished With Errors
  ●  Failed
  ●  Interrupted
  ●  In Progress (Schemas with a job that is running or in planning)
  ●  In Queue
Last Successful LoadtimestampThe date and time of the last successful load job for the physical schema, if any
Last LoadlinkA letter that represents the type of the last or current load job and its start date and time, if any, regardless of the job status. The letter is one of the following:
  ●  F for Full load
  ●  I for Incremental load
  ●  S for Staging
Select the timestamp link, if any, to go to the job details.
Next LoadtimestampThe type and the date and time for the next scheduled load job, if any. If the schema is not part of any load plan, it shows Not Scheduled.
Model StatuslinkThe status of the last or current schema update job (model update job). Hover over it to display the time spent for each phase in the update job. Select it to go to the Model Update Viewer.
For newly added physical schemas (whether created or imported without overwriting existing ones), no model status is available.
If one or more of the physical schema objects have changes that require loading data, there will be a triangle with an exclamation mark next to the status till the required load occurs, whether manually or through a scheduled job.
If the current update job is still in queue because one or more load or update jobs are blocking it, the Model Status (In Queue) will be a link that opens a pop-up that shows a list of these blocking jobs.
Data SizetextThe size of the physical schema data that is loaded into the engine memory
OwnertextThe display name of the user creator
Modified BytextThe display name of the user that last modified the physical schema
More Options → Edit Descriptionmenu linkOpens the Edit Description dialog
More Options → Performance Optimizationmenu linkOpens the Optimize Table Performance dialog
More Options → Change Load Ordermenu linkOpens the Choose Load Order dialog
More Options → Add Load Planmenu linkOpens the New Load Plan dialog
More Options → Sharemenu linkOpens the Share dialog
More Options → Exportmenu linkOpens the Export Schema(s) dialog
More Options → Deletemenu linkOpens the confirmation dialog for the physical schema deletion
Important

Before the 2024.1.0 release, you could use the Schema Manager to create a scheduled job to send email notifications with the status of loading one or more schemas. Starting with the 2024.1.x releases, load plan notifications have replaced schema load notifications, allowing for creating email notifications at the load plan level rather than the schema level. You can use the Scheduler to create load plan notifications.

Important

For physical schemas with dependencies, a physical schema load or update job can block the load and update jobs of dependent physical schemas. Blocked update jobs stay in the In Queue state until all blocking jobs are completed while the blocked load jobs stay in the Load state. You can view a list of jobs that block update jobs. For more details, see View blocking jobs.

Note: Aborted load jobs

Aborted is the status of a load job that is automatically forced to stop after exceeding the configured interruption timeout. Incorta automatically aborts any load job that is interrupted in the extraction phase and exceeds the configured interruption time interval.

Schema Manager actions for physical schemas

With the Schema Manager, you can perform the following actions for physical schemas:

Schema Manager access rights matrix

As the Super User or a user that belongs to a group with the Schema Manager or the SuperRole role, you can access the Schema Manager. You can import physical schemas and create them, whether with the Schema Wizard or the Schema Designer. However, to be able to view, search for, or manage items on this page, you need to have the suitable access rights to them.

The following table shows the required access rights for each action.

Action/Access RightsOwner/EditShareView
Schema → View on list
Schema → Search for or filter
Schema → View details
Schema → View Description
Schema → Add/Edit Description
Schema → Edit details
Schema → Export
Schema → Optimize table performanceView onlyView only
Schema → Manage load orderView onlyView only
Schema → Create load plan
Schema → ShareView only
Schema → Delete
Note
  • When the Cluster Management Console (CMC) admin turns the Enable Super User Mode option on, the Super User and users with the SuperRole can manage all physical schemas regardless of the access rights.
  • Only the Super User tenant administrator can view the list of all items on the Schema Manager regardless of the access rights to them and the Enable Super User mode option.

Create a physical schema with the Schema Wizard

Important: Validation rules for a physical schema name

A physical schema name…

  • Must be unique.
  • Must be between 1 and 250 characters in length.
  • Must begin with an alpha character, lower or upper case.
  • After the first alpha character, can contain zero or more alphanumeric characters in lower, upper, or mixed case.
  • After the first alpha character, can contain zero or more underscore (_) or Dollar sign ($) characters.
  • Besides underscore (_) and Dollar sign ($), cannot contain special characters, symbols, or spaces.
  • Is not case-sensitive. (Starting with 2024.1.x)

Here are the steps to create a physical schema with the Schema Wizard:

  • On the Navigation bar, select Schema.
  • On the Action bar, select + NewSchema Wizard.
  • In (1) Choose a Source, specify the following:
    • In Enter a name, enter the physical schema name.
    • In Select a Datasource, select the external data source.
    • Optionally, create a description.
  • In the Schema Wizard footer, select Next.
  • In (2) Manage Tables, in the Data panel, select the name of the Data Source, and then select the tables, files, or worksheets you want or select the Select All checkbox to select all tables, files, or worksheets depending upon the data source you selected. For each selected table, you can change the table name in Incorta in releases starting 2024.7.x.
  • In the Schema Wizard footer, select Next.
  • In (3) Finalize, in the Schema Wizard footer, select Create Schema.

Create a physical schema with the Schema Designer

Before creating a physical schema with the Schema Designer, review References → Connectors → for specific table property definitions.

Here are the steps to create a physical schema using the Schema Designer:

  • On the Navigation bar, select Schema.
  • On the Action bar, select + NewCreate Schema.
  • In Name, specify the physical schema name, and select Save.
  • In Start adding tables to your physical schema, select one of the following:
    • Schema Wizard: to use the Schema Wizard to select the tables, files, or worksheets you want to add
    • Incorta Analyzer: to use the Analyzer to create a table or aggregated table insight as a physical schema Incorta Analyzer table
    • Incorta SQL: to use Incorta SQL Engine to create an Incorta SQL table by querying objects in the same or other physical schemas.
    • SQL Table: to create a SQL table from different data source types, such as MySQL, Oracle, IBM DB2, Cassandra (Simba), NetSuite Suite Analytics, Presto, SAP Hana, and BigQuery
    • File System: to create a table from a local data file
    • Cassandra: to create a table from an Apache Cassandra data source
    • Cisco Meraki: to create a table from a Cisco Meraki data source
    • Data Lake: to create a table from a Data Lake data source
    • DynamoDB: to create a table from an AWS DynamoDB data source
    • Essbase: to create a table from an Oracle Essbase data source
    • Freshservice: to create a table from a Freshservice data source
    • Google Sheets: to create a table from a Google Sheets data source
    • Kafka: to create a table from an Apache Kafka data source
    • NetSuite: to create a table from a NetSuite Web Service data source
    • OneLogin: to create a table from a OneLogin data source
    • Oracle Cloud Applications: to create a table from an Oracle Cloud Application data source
    • SalesForce: to create a table from a Salesforce data source
    • ServiceNow: to create a table from a ServiceNow data source
    • SharePoint: to create a table from a SharePoint data source
    • Splunk: to create a table from a Splunk data source
    • Zuora: to create a table from a Zuora data source
    • Materialized: to create a Materialized View
    • Alias: to create an alias that references a selected physical schema object, in the same physical schema or other physical schemas
  • In the Data Source dialog, specify the properties for the given object.
  • Select Add.
  • In the Table Editor, in the Table Summary section, enter the physical schema object name.
  • To save your changes, on the Action bar, select Done.

Import a physical schema

You can import one or more physical schemas that exist in the same .zip file. When you import a physical schema, if there is no physical schema that shares the same name with the imported one, a new physical schema is created from this imported one with no version. If you try to import a physical schema that shares the name with an existing one, the import fails. However, you can overwrite this existing physical schema. When you overwrite a physical schema that has saved versions (whether named or not), these versions are not deleted, and a new saved version is added as the current version.

Note

You cannot import a physical schema with the Overwrite existing schema option enabled if another user is editing it in the draft mode nor during a schema update job.

Here are the steps to import a physical schema:

  • On the Navigation bar, select Schema.
  • On the Action bar, select + NewImport Schema.
  • In the Import Schema dialog, optionally select the Overwrite existing schema checkbox to overwrite any existing physical schema that shares the same name with the imported one, if any.
  • Do one of the following:
    • Drag the .zip file that contains the physical schema(s) that you want to import to the Import Schema dialog.
    • Select Click or drag a file here to upload to select the .zip file, and then select Open.
  • Wait until the physical schema(s) and scheduled load job(s), if any, are imported.
  • In the Import Results dialog, review the status of the import operation.
  • Select Close.

The following are the details in the Import Results dialog:

PropertyDescription
NameThe name of the physical schema or scheduled load job
TypeThe type of the imported item: Schema or Scheduler
StatusThe status of the import operation. It can be one of the following:
  ●  Created
  ●  Overwritten
  ●  Failed
If the import operation fails, the reason is displayed. If the physical schema import operation fails, related scheduled load jobs are not imported even if the same physical schema already exists.
Note
  • Importing a schema with its load plans will overwrite the single-schema load plans only.

View a list of physical schemas

  • On the Navigation bar, select Schema.
  • On the Schemas tab, review the list view of physical schemas.

Search for a physical schema

  • On the Navigation bar, select Schema.
  • On the Schemas tab, on the Search bar, enter a search term.
  • Review the list view of physical schemas.

To reset the search, on the Search bar, in the search box, select the remove button (x).

Filter physical schemas by the load status

  • On the Navigation bar, select Schema.
  • On the Schemas tab, in the list view header row, select the filter icon next to Load Status.
  • Select one or more load statuses to filter the physical schemas list view. The load status options are:
    • Succeeded
    • Finished With Errors
    • Failed
    • Interrupted
    • In Progress (Running or in Planning)
    • In Queue
  • Select OK.

Reset the load status filter

  • On the Navigation bar, select Schema.
  • On the Schemas tab, in the list view header row, select the filter icon next to Load Status.
  • Select Reset.

Filter physical schemas by the model update status

  • On the Navigation bar, select Schema.
  • On the Schemas tab, in the list view header row, select the filter icon next to Model Status.
  • Select one or more model update statuses to filter the physical schemas list view. The model update status options are:
    • Succeeded
    • Finished With Errors
    • Failed
    • Interrupted
    • In Progress
    • In Queue
  • Select OK.

Reset the model update status filter

  • On the Navigation bar, select Schema.
  • On the Schemas tab, in the list view header row, select the filter icon next to Model Status.
  • Select Reset.

Edit a physical schema

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema that you own or have edit access rights to, select any part of the row except the Status and Last Load links.
  • In the Schema Designer, you can do the following:
    • view a list of the physical schema objects and joins
    • add, edit, or remove physical schema objects
    • add, edit, or remove joins between the physical schema objects
    • load data for the physical schema
    • load data for a selected physical schema object
    • view or manage load jobs
    • open the Table Editor for a selected physical schema object
    • switch to the Diagram view
    • open the Join Editor for a select join
    • explore physical schema data
    • add or edit a physical schema description
    • optimize performance for the physical schema objects
    • change the load order of the physical schema objects
    • create a load plan
    • share a physical schema

View physical schema details

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema, select any part of the row except the Status and Last Load links.
  • In the Schema Designer, you can do the following:
    • view load jobs
    • view physical schema objects and joins
    • view physical schema object details, such as the columns, formula columns, and load filters
    • explore physical schema data
    • view the physical schema diagram
    • view or edit the share options for the physical schema depending upon the access rights
    • view the physical schema description, objects load order, and performance optimization settings for the physical schema objects

For more information, refer to Tools → Schema Designer

View the load job details for a physical schema

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema, select the Load Status or Last Load link.
  • In the Load Job Details Viewer, review the details of the load jobs for the physical schema.
Note

You can track all load jobs and load plans related to a physical schema on the Schema Manager → Load Jobs.

View the physical schema update jobs

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema, select the Model Status link.
  • In the Model Update Viewer, review the details of the physical schema update jobs.

View blocking jobs

When an update job or a load job is in the commit phase or the load and post-load phase, respectively, it blocks the update and load jobs on dependent physical schemas. A dependent physical schema is one that has a cross join relationship with another physical schema. The blocked update jobs stay in the In Queue state till all blocking jobs are completed while the load jobs stay in the Load state. As a schema manager user, you can view a list of jobs that block update jobs in the Schema Manager and the Model Update Viewer. This list shows the blocking physical schema, the type of the blocking job, load or update, and the current status of each blocking job. You can also access the details of the blocking job in the Load Job Details Viewer or the Model Update Viewer according to the type of the blocking job.

Here are the steps to view the list of blocking jobs of an update job in the Schema Manager:

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema that has a blocked update job, in Model Status, select the In Queue link. A pop-up dialog appears with a list of the blocking jobs.
  • To view the details of a blocking job, in the pop-up dialog, select See Job Details.

Add or edit a physical schema description

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
  • Select Edit Description.
  • In the Edit Description dialog, enter or update the physical schema description. You can enter up to 500 alphanumeric characters, including spaces and special characters.
  • Select Save.

Optimize performance for physical schema objects

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
  • Select Performance Optimization.
  • In the Optimize Table Performance dialog, in the objects list, for each object you want to optimize the performance, select the checkbox, and for each object that you do not want to optimize the performance, clear the checkbox.
  • Select Save.
Warning

You must enable performance optimization for Incorta Analyzer tables and Incorta SQL tables to avoid load failure.

Note

For objects that you enable for performance optimization, the Loader Service will load their files from Shared Storage (Staging) into the Engine memory. For unselected objects, the Loader Service will load data only into Shared Storage, which can slow the speed of creating insights; however, this will reduce the load time.

Change the load order of physical schema objects

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
  • Select Change Load Order.
  • In the Choose Load Order dialog, you can do the following to manage the table groups before changing their load order:
    • For each table or Materialized View that you want to add to a new load group, select the checkbox, and then select Add to New Group.
    • For each table or Materialized View that you want to add to an existing load group, select the checkbox, and then select Add to Existing Group.
    • To reset table grouping and order, select Revert to Default.
  • To set the order of a table group, select the up arrow (to the right of the group name) to move the group upwards, or select the down arrow (to the right of the group name) to move the group downwards. Repeat this action until you order the table groups as required.
  • Select Save.

Share a physical schema

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
  • Select Share.
  • In the Share dialog, you can do the following:
    • remove an existing user or group
    • modify the existing access rights for a user or group
    • add a new user or group with the required access rights

Export a physical schema

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
  • Select Export.
  • In the Export Schema(s) dialog, optionally select the Include load plans checkbox to include scheduled load jobs or load plans related to the selected physical schema in the export operation.
  • Select Export. The exported physical schema .zip file is downloaded to the Downloads folder.
Note
  • When you export a physical schema, you export the current saved version only. No drafts or old saved versions are exported. To export a previous saved version, use the Schema Designer, for more details, refer to Export a saved version.
  • Exporting a schema with its load plans will include only single-schema load plans that include this schema.

Export one or more physical schemas

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for each physical schema that you want to export, select the checkbox.
  • On the Search bar, next to Selected, select More Options (⋮ vertical ellipsis).
  • Select Export.
  • In the Export Schema(s) dialog, optionally select the Include load plans checkbox to include scheduled load jobs or load plans related to the selected physical schema(s) in the export operation.
  • Select Export. The exported physical schema(s) .zip file is downloaded to the Downloads folder.

Delete a physical schema

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for a given physical schema row, select More Options (⋮ vertical ellipsis).
  • Select Delete.
  • In the dialog, select Delete.
Notes
  • When you delete a schema, the history of scheduled load plans of this schema will be maintained in the Schema Manager > Load Jobs page.
  • You cannot delete a schema if another user is editing it in the draft mode.
  • You cannot delete a schema if it is part of a running load job, whether scheduled or manual.
  • When you delete a schema that exists in a single-schema load plan, the load plan will be automatically deleted along with its scheduled runs.
  • When you delete a schema that exists in a multi-schema load plan:
    • If the schema exists in a load group with other schemas, the schema will be removed from the group.
    • If the schema exists alone in a load group, the group will be removed from the load plan.

Delete one or more physical schemas

  • On the Navigation bar, select Schema.
  • In the list view of physical schemas, for each physical schema that you want to delete, select the checkbox.
  • On the Search bar, next to Selected, select More Options (⋮ vertical ellipsis).
  • Select Delete.
  • In the dialog, select Delete.