Tools → Load Job Details Viewer
About the Load Job Details Viewer
The Load Job Details Viewer is a new tool available to schema managers starting 2022.12.0 to review the status and history of a load job. Additionally, it lists the history of load jobs related to the selected one. Starting with 2023.4.0, the Load Job Details Viewer replaced the Load Job Viewer.
Starting with 2023.7.0, you can organize schemas in a load plan into separate groups. The Loader Service simultaneously loads all schemas in the first group, then starts loading the next group, and so on. The Loader Service executes each group as a separate load job. The Load Job Details Viewer separately shows the details of each group.
Load Job Details Viewer access rights
A Super User or a user that belongs to a group with the SuperRole or Schema Manager role can access the Load Job Details Viewer to view the details of load jobs related to schemas that the user owns or has access rights to. When the Cluster Management Console (CMC) admin turns the Enable Super User Mode option on, the Super User and users with the SuperRole can access the Load Job Details Viewer of all load jobs regardless of the access rights to the related schemas.
To access the Load Job Details Viewer, in the navigation bar, select Schema, select the Load Jobs tab, and then select a load job. By default, the Load Job Details Viewer shows the schemas, objects, and details of the load job you select on the Schema Manager → Load Jobs.
Starting with 2023.4.0, you can access the Load Job Details Viewer as follows:
- In the navigation bar, select Schema, and then on the Schemas tab, select the status or time of the latest or current load job.
- In the navigation bar, select Schema, and then on the Schemas tab, select a schema. On the Schema Designer, select the status or time of the latest or current load job.
- On the Home page, select the Job Details of a failed, interrupted, or finished with errors job.
In these cases, if the latest or current load job is related to a single-group load plan, the Load Job Details Viewer by default filters the schemas list to show the relevant schema only. And if the latest or current load job is related to a multi-group load plan, the Load Job Details Viewer by default expands the group with the selected schema, collapses other groups, and filters the group schemas to show the relevant schema only. You can remove or update the filters and expand other groups to view other schemas in the load plan.
Load Job Details Viewer vs Load Job Viewer
Although there is another tool that shows the details and history of a load job, that is, the Load Job Viewer, there are differences between the two tools.
The following table shows the differences and similarities between the two tools.
| Property | Load Job Details Viewer | Load Job Viewer |
|---|---|---|
| Accessed | By selecting a load job in the Schema Manager → Load Jobs Starting with 2023.4.0, you can access it by selecting the status or time of the latest or current load job on the Schema Manager → Schemas or the Schema Designer or selecting the Job Details of a failed, interrupted, or finished with errors job on the Home page. | Before 2023.4.0, by selecting the load status or load time in the Schema Manager → Schemas or Schema Designer. |
| Single-schema jobs | ✓ | ✓ |
| Multi-schema jobs | ✓ | |
| Scheduled job | ✓ | ✓ |
| Manual job | ✓ | ✓ |
| Differentiates between manual and scheduled load jobs | ✓ | |
| Shows who invoked the job | ✓ | |
| Shows the tool used to invoke the job | Shows an empty value in the case of a scheduled load job. However, it doesn’t differentiate between the tools used to invoke manual load jobs, whether the application UI or API calls. It shows the user name in all cases. | |
| Shows the total rows per object | ✓ | |
| Allows downloading the rejected rows report | ✓ (starting 2023.1.0) | ✓ |
- A manual load job is one that users invoke either using the application UI (via the Schema Designer or the Scheduler > Load Plans starting 2024.1.0) or an API endpoint.
- A scheduled load job is one that the Scheduler runs depending on load plans with one or more physical schemas.
Anatomy
The Load Job Details Viewer consists of the following:
- Page header
- Load job history
- Load job details
Page header
The page header shows the following:
- Breadcrumb navigation
- The schema or load plan name
- The last sync time of the load jobs and details, and a Refresh button to manually refresh them
Load job history
This part shows the history of load jobs related to the job that you select on the Load Jobs list.
- If you select a load job of a scheduled load plan with multiple schemas, this list shows all the load jobs of this load plan, that is, the runs of this scheduled load plan.
- If you select a load job that you or another user manually started for a specific physical schema (via the application UI or API calls) or a load job of a scheduled load plan with a single schema, the list shows:
- All manual load jobs of this physical schema, including those that other users invoked, whether via the application UI or API calls.
- Load jobs of all load plans with this single schema.
The load job history list shows the following details for each load job:
- Status (icon): You can filter the job list by the job status.
- Job Starting Time: The job start time. By default, the list is sorted in descending order according to the job start time.
- Duration: The time elapsed during the load job. You can sort the list according to the load job duration. The duration of each stage in the load job is visualized using a colored bar. If the load plan includes multiple groups, the colored bar can be repeated according to the number of groups executed in the load plan as each group is considered as a separate load job.
If a group load job fails due to a Loader Service issue, and the Loader Service retries to load the same group again, this group will appear twice on the object list, and the number of groups will be increased by one.
The history list shows the recent 50 load jobs, you can navigate to earlier jobs using the page controls. Additionally, you can collapse the history pane to allow more space for the job details part.
- Before 2023.7.0, when you delete a physical schema, the job history of this schema is completely deleted, whether manually or scheduled load jobs.
- Starting with 2023.7.0, when you delete a physical schema, the following applies:
- Manually started jobs will be deleted and won't appear on the list.
- Load jobs of single-schema load plans with this schema won't be deleted.
- Load jobs of multi-schema load plans with this schema won't be deleted.
Load job details
This part shows the details of the selected load job, including the physical schemas and objects. By default, this part shows the details of the load job you selected on the Load Jobs list or the latest load plan if you accessed the Load Job Details Viewer from other pages, such as the Schema Manager, Schema Designer, or the Home page. However, you can select another job to show its details.
This part contains a summary of the selected load job. The summary includes the start time of the selected load job, its status, type: manual or scheduled, and the load plan name (starting from 2024.1.0) if the selected load job is part of the execution of a load plan regardless of the execution type.
You can search for objects and physical schemas in the selected load job. You can also control the columns to display on the list; the column selection is cached in the browser.
Starting with 2023.7.0, a load plan can have multiple schemas gathered in groups. If the selected load plan contains multiple groups, The summary section shows the following depending on the status of the selected load plan:
- In the case of a running, finished with error, or successful load plan, this section shows the total number of groups and schemas in the load plan. The object list shows only running or executed groups with their schemas.
- In the case of an interrupted or failed load plan, this section shows the number of only groups and schemas that have been executed.
Additionally, the object list shows a summary row for each group and the details of all schemas and objects in the group.
This page shows the following details for each object in the load job.
| Column | Description |
|---|---|
| Summary | This row shows the following: ● The total duration of the load job ● The duration of each stage ● The number of schemas and objects in the load job ● An information icon that shows the ID of the load job and the Loader Service that ran it ● The total number of rejected rows, if any (Starting 2024.1.0) If the selected load job is related to the execution of a multi-group load plan, these details are displayed per group in addition to the following: ● The group number ● The start time of loading the group schemas ● The status of loading the group |
| Schema Name | The schema name |
| Table Name | The object name |
| Schema / Table | Available starting with 2024.1.0, replacing the Schema Name and Table Name columns. Select the schema name to expand or collapse objects in the schema. This list is collapsed by default and is sorted in ascending order by the schema name, and within each schema, objects are sorted alphabetically. |
| Load Type | The load type per schema or object: ● F for full load ● I for incremental load ● S for load from staging |
| Extraction/Enrichment > Start | The start time of the object extraction or enrichment |
| Extraction/Enrichment > Duration | The elapsed time during the object extraction or enrichment |
| Extraction/Enrichment > Extracted | The number of rows extracted or enriched for this schema or object during this load job, including the rejected rows |
| Extraction/Enrichment > Rejected | The number of rejected rows during the extraction or enrichment per schema or object |
| Deduplication > Start | Available starting with 2024.1.0. The start time of the deduplication phase, which includes the PK-index creation and Parquet compaction processes |
| Deduplication > Duration | Available starting with 2024.1.0. The elapsed time during the deduplication phase |
| Load/Post-Load > Start | The start time of loading the object into the Engine memory. All Load/Post-Load columns will show no data for non-performance-optimized objects. |
| Load/Post-Load > Duration | The elapsed time to complete the Load/Post-Load calculations for the respective object |
| Load/Post-Load > Loaded | The number of rows loaded during the selected load job per schema or object including those: ● Loaded from the extracted/enriched rows. ● With duplicates in the memory. ● That failed to load during previous load jobs and were loaded during the selected load job. |
| Load/Post-Load > Total Rows | The total number of unique rows loaded into the Engine memory for the respective schema or object after the selected load job, including rows loaded during previous load jobs and the selected load job. This number might be different from the current total number of rows loaded for the related schema or object. For non-performance-optimized objects, the total rows are 0 as no rows from this object are loaded into the Engine memory. |
| Send to Destination > Start | The start time of sending the object to configured data destination. |
| Send to Destination > Duration | The elapsed time to complete the Send to Destination calculations for the respective object. |
| Send to Destination > Sent Rows | The number of rows sent during the selected load job per schema or object |
| Send to Destination > Total Rows | The total number of rows at the data destination per schema or object after the rows were sent. You can always compare the total numbers in the data destination with the total number of rows in each loading phase. |
| Status | The load status per schema or object |
By default, the details list is sorted per group in ascending order by the schema name. You can use all columns to sort this list per group, except for the Load Type column. Sorting is applied to all rows in the load group. As a result, the group schemas are expanded, and the schema name might appear multiple times on the sorted list. When you select the header of a column, the list is sorted in ascending order by this column; select it again to sort it in descending order. Selecting the same column header for the third time resets the list sorting.
You can use the Schema/Table, Load Type, and Status columns to filter out the schemas and objects that appear on this list. Starting with 2024.1.0, applied filters are preserved when refreshing the details list.
The total number of rows differs according to the loading type whether it is incremental, full, or staging load. The load job sends the data to destination only if you are using full or incremental load.
Also, the Send to Destination section is only available if you have a data destination set for one or more schemas in the load job.
- If a multi-group load job fails or finishes with errors, when you select the status, the error message shows the status of each group that causes the errors. When you select the status of the group on the objects list, the error message shows the error per object with a link to view the error details.
- If a multi-group load job is interrupted:
- The job status of the whole load plan will be Interrupted.
- The job status of the group that is running when you stop the job will be Interrupted.
- Finished groups in the load plan will not be impacted by stopping the running job and will have their own status.
- Groups that have not started yet will not appear on the Load Job Details Viewer.