Tools → Schema Designer → Load Jobs
About the Schema Designer Load Jobs
You can use the Schema Designer to perform or schedule load jobs of all objects in the physical schema or to load a specific object.
The Summary section in the Schema Designer shows the status of the last load job or the stage of the current load job. You can review the details of the current or previous load jobs in the Load Job Viewer or the Load Job Details Viewer (starting with 2023.4.0).
Incorta supports different types and strategies for loading data. To learn more about loading data into Incorta, refer to References → Data Ingestion and Loading.
Changes that require data load
Some updates you make to the physical schema objects require loading data, either load from source (full load) or load from staging, to ensure data consistency.
The following are the updates that require load from staging:
- Adding, updating, or removing the load filter
- Enabling performance optimization for non-optimized objects
The following are the updates that require full load:
- Adding a new physical schema table or materialized view
- Changing the data type of a physical schema table column or MV column
- Changing the source of a materialized view or physical schema table, whether by selecting another source file in the Data Source properties dialog or editing the query
- Adding or changing a key column (changing the column function from key to dimension or measure and vice versa) in a physical schema table or MV
- Adding a new physical schema table column
- Adding a new MV column
- Changing the object type, for example, changing a physical schema table to an Incorta Analyzer table or MV
- Removing a physical schema table column or a materialized view column that functions as a key
- Changing the encryption status of one or more columns in a physical schema table or materialized view
Incorta does not recommend running concurrent schema model update jobs and load jobs on the same schema or dependent schemas as this may result in errors or inaccurate data.
Schema Designer actions for load jobs
With the Schema Designer, you can perform the following actions for data load jobs
- Change the load order for physical schema objects
- Perform a full load of the physical schema data
- Perform an incremental load of the physical schema data
- Load physical schema data from staging
- Schedule a physical schema load job
- Create a load plan (Available starting 2022.11.0)
- Perform a load of a physical schema object
- Review the status of a load job
- Create a physical schema load email notification
Before the 2024.1.0 release, you could use the Schema Designer to create a scheduled job to send email notifications with the status of loading a schema. Starting with this release, load plan notifications have replaced schema load notifications, allowing for creating email notifications at the load plan level rather than the schema level. You can use the Scheduler to create load plan notifications.
Change the load order for physical schema objects
- In the Navigation bar, select Schema.
- In the list view of schemas, select a given physical schema.
- In the Schema Designer, select Settings (gear icon).
- Select Change Load Order.
- In the Choose Load Order dialog, you can do the following to manage the table groups before changing their load order:
- For each table (or materialized view) that you want to add to a new load group, select the check box, and then select Add to New Group.
- For each table (or materialized view) that you want to add to an existing load group, select the check box, and then select Add to Existing Group.
- To reset table grouping and order, select Revert to Default.
- To set the order of a table group, select the up arrow (to the right of the group name) to move the group upwards, or select the down arrow (to the right of the group name) to move the group downwards. Repeat this action until you order the table groups as required.
- Select Save.
Perform a full load of the physical schema data
- In the Schema Designer, in the Action bar, select Load → Full Load.
- In the dialog, select Load.
Perform an incremental load of the physical schema data
- In the Schema Designer, in the Action bar, select Load → Incremental Load.
- In the dialog, select Load.
Load physical schema data from staging
- In the Schema Designer, in the Action bar, select Load → Staging Load.
- In the dialog, select Load.
Schedule a physical schema load job
Starting with 2022.11.0, the Add Load Plan option has replaced the Add Schema Load option, allowing for creating load plans to load one or more physical schemas.
Here are the steps to schedule a physical schema load job in releases before 2022.11.0.
- In the Schema Designer, in the Action bar, select Load → Schedule Load Job.
- In the Add Schema Load dialog, in Schema Load Name, enter the name of the scheduled load job. You can enter up to 255 alphanumeric characters, including spaces, special characters, and emojis.
- In Description, optionally enter a description for the scheduled load job. You can enter up to 255 alphanumeric characters, including spaces, special characters, and emojis.
- The Schema box shows the name of the current physical schema.
- In Load Type, select the load type. The available options are:
- Full
- Incremental (default)
- Staging
- In the Recur Every section, specify the frequency and recurrences of this job.
- Enter a number.
- Select the time measure unit: minutes, hours, days, or months.
- If you select Week, specify the days to run the job on.
- If you select Month(s), select the day or weekday you want to run the job, for example, the second Monday of the month.
- Select the No Recurrence check box to run this job only once.
- In the At section, specify the time to run this job. Select the time and time zone.
- Specify the validity period of the scheduled load job:
- If the job is recurring, in the From section, do one of the following:
- Select the start date and keep the Does not end check box selected to run it forever.
- Select the start date, clear the Does not end check box, and then select the end date to run the job during this period.
- If the job is not recurring, in the On section, select the date to run this job.
- If the job is recurring, in the From section, do one of the following:
- Select Add.
The scheduled load jobs appear in the Scheduler, under Schema Loads. A user with the appropriate access rights to the Scheduler can manage these jobs, in addition to creating new jobs.
Create a load plan
This feature is available starting with 2022.11.0. To be able to create a load plan with multiple schemas, the multi-schema loading feature must be enabled on your cluster. This feature is always enabled starting with 2023.7.0. You can add to a load plan only schemas that you own or have edit access rights to.
In addition to the Schema Designer, you can create a load plan from the Schema Manager and the Scheduler.
Scheduled load plans appear in the Scheduler under Load Plans. A user with the appropriate access rights to the Scheduler can manage these plans.
You can create a load plan from the Schema Designer by applying the following steps in releases before 2023.7.0:
- In the Schema Designer, in the Action bar, select Load → Add Load Job.
- In the Add Load Plan dialog, specify the load plan details.
- Select Add.
The following table shows the details of a load plan.
| Property | Control | Description |
|---|---|---|
| Name | text box | Enter a unique name for the load plan you are creating. |
| Description | text box | Enter a description for the load plan. |
| Schemas/Schema | list/ dropdown list | Select the schema or schemas that you want to schedule a job to load. The list displays the schemas that you can edit. The Schemas list is available only when the multi-schema loading feature is enabled. If the list contains more than 20 schemas, page controls are available to navigate the list. |
| Schemas > Search Schemas | text box | Available only when the multi-schema loading feature is enabled. Search for the schemas you want to include in the load plan. |
| Schemas > Default Load Type | dropdown list | Available only when the multi-schema loading feature is enabled. Specify the default load type before selecting the required schemas. Specifying a default load type doesn’t affect the load type you specify for the selected schemas. To apply the default load type to the selected schemas, select Reset. |
| Schemas > select all | checkbox | Select all schemas that currently appear on the list. ● If the list contains multiple pages, you select only schemas on this page. ● If you search for schemas, you select the schemas that match your search. ● If the search result contains multiple pages, you select the schemas on the current page. |
| Load Type | dropdown list | Select the load type you want to perform. The types are ● Full ● Incremental (default) ● Staging When creating a multi-schema load plan, you can determine the load type per schema. You can also apply a specific type to all schemas. |
| Recur Every | increments and dropdown list | Select the recurrence of the job by selecting an interval of minutes, hours, days, weeks, or months. This option isn’t available when you select the No Recurrence checkbox. |
| No Recurrence | checkbox | Select the checkbox to schedule the job for one time only. This option is unselected by default. |
| At | time and time zone selection | Select the time and time zone to run the load plan job. Select Now to immediately run the job. |
| Between | checkbox | Select the checkbox to schedule the load plan job to run within a specific period. You have to define the start time, end time, and timezone. |
| Every | increments and dropdown list | Select the recurrence of the load plan job by selecting an interval of minutes or hours. This option is available only when you select the Between checkbox. |
| From / On | calendar | Select the date range of the load plan job. The field name changes to On when you select the No Recurrence checkbox. |
| Does not end | checkbox | Select this checkbox to disable the end date of the load plan job. It isn't available when you select the No Recurrence checkbox. |
| Add | button | Select to save the dialog and add the scheduled job to the list. |
| Cancel | button | Select to discard what you have entered and close the dialog. |
Create a load plan in 2023.7.0 and later
The following are the steps and details of creating a load plan from the Schema Designer starting with 2023.7.0.
- In the Schema Designer, in the Action bar, select Load → Add Load Job. The New Load Plan dialog opens.
- Enter a name for the load plan and optionally enter a description.
- Specify the default load type for the schemas you will add to the load plan.
- Under Select Schema, drag one or more schemas to the Selected Schemas list.
- To select multiple schemas, use the Command or Ctrl key while selecting the schemas. You can add a schema only once in the load plan.
- Optionally, change the load type for each schema.
- To remove a schema from the list, select the remove button (x icon).
- Optionally, select Set sequential loading groups to add a new group and drag schemas to it.
- To add more groups, select New Group.
- To remove all schemas from the group, select More Options (⋮ vertical ellipsis icon), and then select Clear All.
- To move a schema from one group to another, drag the schema from a group to another.
- To delete a group, select More Options (⋮ vertical ellipsis icon), and then select Delete
- Select Save.
- Under Scheduler, specify the scheduling options of the load plan, and then select Save.
- To save the load plan without setting a schedule to run it, select Schedule later.
- Select Close.
The default load type option is available starting with 2024.1.0. Whenever you add a new schema to the load plan, it inherits the default load type. If you change the default load type, it will not affect the schemas that you have already added.
Perform a load of a physical schema object
- In the Schema Designer, in the Tables tab, for a given physical schema object, select More Options (⋮ vertical ellipsis).
- Do one of the following:
- Select Load Table to start a full load job.
- Select Load from Staging to start a load from staging job.
- In the dialog, select Load.
Review the status of a load job
- In the Schema Designer, in the Summary section, select the Last Load Status link.
- In the Load Job Viewer/Load Job Details Viewer, review the load job status.
Create a physical schema load email notification
This feature is not available starting with 2024.1.0. Load plan notifications replaced it. You can use the Scheduler to create load plan notifications.
Notifications require a tenant email configuration for an outgoing email server using SMTP or EWS in the Cluster Management Console (CMC).
The notifications that you create appear on the Scheduler, under Schema Notifications. A user with the appropriate access rights to the Scheduler can manage these notifications and create new notifications.
Here are the steps to create an email notification for a physical schema:
- In the Navigation bar, select Schema.
- In the list view of schemas, select a given physical schema.
- In the Schema Designer, in the Action bar, select Settings (gear icon).
- Select Create Email Notification.
- In the Create Email Notification dialog, enter the details of the email notification. For more information, see Email notification details.
- Select Done.
Email notification details
The following table shows the details you need to specify for a load email notification:
| Property | Control | Description |
|---|---|---|
| Notification Name | text box | Enter the name of the new email notification. The length of the notification name must be between 1 and 255 alphanumeric characters, including spaces and special characters. |
| Notify On → Success | checkbox | Select the checkbox to send the notification email when the load successfully completes |
| Notify On → Failure | checkbox | Select the checkbox to send the notification email when the load fails |
| Select Schema(s) → Search | text box | Enter a search term to search for specific physical schemas to create load notification for. |
| Select Schema(s) → Select All | checkbox | Select the checkbox to select all physical schemas you have access to or all physical schema(s) in the search result list |
| Select Schema(s) → Schemas list | list/checkbox | Select the checkbox for each physical schema you want to include in the load notification. Only physical schemas that you own or have access rights to appear on the list. |
| Recipients → Search Names, Emails, and Groups | text box | ● Enter a search term (for example, an email address, user name, group name) to search for users or groups in the tenant. ● Enter the email address to select an external recipient. |
| Recipients → List | list | For each recipient, select V (down arrow) to the right of the recipient name or email, and then select where the given recipient should go to: the To list, Cc list, or Bcc list. |
| Body | text box | Optionally, enter the email body text. The body text you enter appears in the notification email before an auto-generated text. You can enter up to 4000 alphanumeric characters, including spaces and special characters. |
When you create an email notification that contains multiple physical schemas, the recipient(s) receives a separate email for each physical schema. Recipients on the Bcc list do not appear on the received email, while those on the To and Cc lists do appear.
The notification email contains the following:
- the email body text you entered, if any
- the notification name
- the display name of the notification user creator
- the schema name
- the schema load status
- the load date and time
- the load duration
- a link to the Load Job Viewer that contains the schema load job summary and details