Tools → Inspector Tool

About the Inspector Tool

For a given tenant, the Inspector Tool checks the lineage references of Incorta metadata objects including tables, schemas, business schemas, business schema views, dashboards, and session variables. It also checks for inconsistencies and validation errors in joins, tables, views, formulas, and dashboards.

Because the tool uses Incorta itself for reporting issues, the Inspector Tool consists of several components:

• a Cluster Management Console (CMC) scheduled job that generates .csv files in a given tenant directory as a local data files
• the InspectorMetadata schema
• the incortaInspector business schema
• the InspectorTool dashboard folder and related dashboards

About an Inspector job

An Inspector job is a scheduled job in the CMC. A CMC Administrator creates an Inspector job.

For the selected tenant in the cluster, an Inspector job performs the following:

• Creates a tenant export
• Creates the related .csv files in the tenant data directory
• Writes the inspector.log file

About the Inspector Tool dashboards

You can use the Inspector dashboards to identify specific issues with your tenant. Here are some examples:

• An invalid join due to mismatched data types or unsupported data types
• A join with a missing table
• A join with a missing column
• A join on a parent table column that is not a key column
• A join on a child table with multiple parent tables
• A join using an invalid formula column
• A join on a formula column that references columns in two or more schemas
• A cyclical join between two or more tables such as A > B > C > D > A that can be resolved with a table alias
• Between two tables, identify multiple join paths
• A table enabled for incremental loads but without incremental logic specified
• A table enabled for incremental loads with incremental logic specified but no key column specified
• A table with a runtime security filter that references a missing session variable or the session variable has a missing definition
• An alias table with no existing base reference table
• An alias table out of sync with the existing base reference table
• A formula that refers to a column that does not exist
• A formula that references columns in two or more schemas
• A business schema view that references a column in a table that does not exist
• A session variable that references another session variable that does not exist
• A dashboard that references a missing session variable
• A dashboard that references a missing table or business schema view column

Identified issues have a severity rating. You can also use dashboards to identify potential performance concerns.

There are two folders in the Inspector Tools dashboard folder: Inspector and Metadata.

Inspector folder

Here is a summary of the dashboards available with the Inspector folder:

• 0- Run status
• 1- Validation UseCases
• 2- Unused Entities
• 3- Schemas Details
• 4- Dashboards Lineage Summary
• 5- Tables Used in Business Views
• 6- Tables Used In Materialized Views

0- Run status

This dashboard captures runtimes of the Inspector Tool. In doing so, the dashboard captures basic counts for the various objects within a tenant for each runtime. The insights are:

• Tenant Validation
• Tenant_Dashboards
• Tenant_Joins
• Tenant_Schemas
• Tenant_BusinessSchemas
• Tenant_MVs

1- Validation UseCases

The Validation UseCases dashboard identifies errors in the tenant such as broken references, duplicates, and invalid formulas. Employ the dashboard filters to quickly identify high severity issues. Focus on fixing all high severity (1 & 2) issues. The dashboard insights are:

• Validation UseCases
• Summary
• Errors & Warnings
• Error & Warning Codes
• Errors & Warnings Per Schema
• Errors Details
• Warnings Details

2- Data Lineage

Use the dashboard to get a quick view of data elements or calculations that could be moved closer to the source. Employ the dashboard filter to help identify the lineage relationships of all the tenant's dashboards. The dashboard insights are:

• Summary
• Dashboard with Insights
• Lineage Data

2- Unused Entities

This dashboard identifies columns and business schema columns that are not used in dashboards. You may consider removing unused columns from business schema or schemas. Removing unused columns can help improve schema load times, reduce storage needs, and free memory. The dashboard insights are:

• Unused Tables
• Unused Columns
• Unused business schema columns

2- Impact Analysis

This dashboard provides the impact analysis between sources, dashboards, and usage details. It also identifies the source type with its details and dashboards source schema and which dashboards they are used in. The dashboard insights are:

• Sources
• Dashboards
• Detailed Usage

3- Schemas Details

The Schemas Details dashboard provides a detailed view of all the schemas in a tenant, including the tables, joins, and columns. The dashboard insights are:

• Schemas Summary
• Tables
• Joins
• Columns

4- Audit Summary

The audit summary dashboard provides a list of the top used dashboards along with the number of visits and the average time taken to view the dashboard and the number of users who view them. The dashboard insight is:

• Top 20 Used Dashboards

5- Tables Used in Business Views

Use this dashboard to identify all the business schemas in a tenant, the view names, and the columns. The dashboard insights are:

• Summary
• List of Business Schemas
• Business Schema Summary

6- Tables Used In Materialized Views

Use this dashboard to identify all the materialized views in a tenant, including any referenced tables within the code or script of the materialized view. The dashboard insight is:

• MV summary

Metadata folder

Here is a summary of the dashboards available in the Metadata folder:

• 0. Environment - Tenant Overview
• 1. Dashboard - User Analytics (BI Audit)
• 2. Job - Load Summary (Schemas-Tables)
• 3. Job Execution Analytics (within one day)
• 4. Dashboards
• 5. Objects Shared with Users
• 6. Security
• 7. User Actions
• 8. Schema-Table Metrics
• 9. SQL Query-User Analytics (BI SQLi_Audit)

0. Environment - Tenant Overview

Use this dashboard, which consists of two tabs: Tenants-Users and Objects, to know details about your tenant, such as tenant's summary, users, user groups, and roles. It also includes details about objects within a tenant, such as physical schemas, business schemas, and other objects. The following tables show the insights available in each tab of the dashboard:

1. Dashboard - User Analytics (BI Audit)

Use this dashboard, which consists of two tabs: User Engagement and Query Performance, to know the answer to multiple questions about dashboards and how users interact with Incorta, as well as tenants’ properties. This dashboard also includes insights about query performance.

The following tables show the insights available in each tab of the dashboard:

2. Job - Load Summary (Schemas-Tables)

Use this dashboard to know details about jobs and loads, such as job/load statuses and types. The following insights are available in this dashboard:

• Job Summary
• Job/Load Status (schema level)
• Service Node Used
• Job Type
• Schema Loads by Frequency/Duration
• Schema Jobs/Loads by Month/Day/Hour
• Schema Load Summary
• Job Details (Schema Loads)

3. Job Execution Analytics (within one day)

Use this dashboard, which consists of two tabs: Daily Analytics and Job GANTT Chart, to view insights about jobs and loads for today's date. Also, you can view details in a Gantt chart.

The following tables show the insights available in each tab of the dashboard:

4. Dashboards

Use this dashboard to view insights to show you details about dashboards in your Incorta platform, such as the number of dashboards, number of folders, and other details. The following insights are available in this dashboard:

• # dashboards
• # folders
• Dashboards by Owner
• Dashboards Last Modified By
• Dashboard Details

5. Objects Shared with Users

Use this dashboard, which consists of two tabs: Dashboards-Folders and Other Objects, to view insights about different objects that have been shared with different Incorta users and their permission types to these objects. The first tab is specific to dashboards and folders, while the other covers physical schemas, business schemas, and other objects.

The following tables show the insights available in each tab of the dashboard:

6. Security (Users-Groups-Roles)

Use this dashboard to view insights about the created users in Incorta per tenant, their emails, user groups, and other details. Also, you can view the objects shared per user.

The following insights are available in this dashboard:

• Summary
• Tenant
• User Source-email domain
• User Authentication Type
• Groups
• Group Members
• Role-Group Relationships
• User Details
• Objects Shared by User

7. User Actions

Use this dashboard, which consists of two tabs: Actions-Metadata and Actions-Audit Data, to check insights about metadata analysis, such as actions done by users and by objects. In addition, you can view audit insights.

The following tables show the insights available in each tab of the dashboard:

8. Schema-Table Metrics

Use this dashboard to view insights about schemas and tables available in your Incorta cluster.

The following insights are available in this dashboard:

• Summary Metrics
• Tenant
• Rows by Tenant-Schema-Table

9. SQL Query-User Analytics (BI SQLi_Audit)

Use this dashboard, which contains three tabs: SQLi Summary, SQLi Trends, and SQLi - Query Details, to view insights about queries that have either failed or been successful. You can view insights about these queries' performance, handlers, and details.

The following tables show the insights available in each tab of the dashboard:

Create an Inspector job

A CMC Administrator creates an Inspector job for a given tenant.

There are two ways to create an Inspector job in the CMC:

• Create an Inspector job in the CMC Scheduler
• Create an Inspector job to execute immediately

Create an Inspector job in the CMC scheduler

A newly created Inspector job is by default an Active job. Here are the steps to create an Inspector job:

• As the CMC Administrator, sign in to the CMC.
• In the Navigation bar, select Scheduler.
• In the Inspector tab, in Cluster Name, select a cluster.
• In the List view, in the Action Menu, select +.
• In the Add a new job dialog, specify the following:
• Name
• Tenant
• Specify the schedule expression as follows:
• Every in terms of numeric frequency, time intervals, and recurrence
• Starting at in terms of a set time and GMT
• From in terms of a Start Date, optional End Date, and Does Not End selection
• Select Add job.

Create an Inspector job to execute immediately

For a given tenant, you can execute an Inspector job immediately. Follow these steps:

• As the CMC Administrator, sign in to the CMC.
• In the Navigation bar, select Clusters.
• In the cluster list, select a cluster name.
• In the canvas tabs, select Tenants.
• In the list view, for a tenant row, select the Kebab icon.
• In the More Options menus, select Execute Inspector now to immediately execute an Inspector job.

Download the Inspector Tool files

A CMC Administrator downloads the related Inspector Tool schema, business schema, and dashboards files for all tenants from here.

Or, you can follow these steps to download schema, business schema, and dashboards:

• As the CMC Administrator, sign in to the CMC.
• In the Navigation bar, select Scheduler.
• In the Inspector tab, select the information icon.
• In the tooltip, select the download link.
• Select Yes, I Agree + Download File.

A folder named Inspector is downloaded that contains the following .zip files:

• business_schema.zip
• dashboards.zip
• schema.zip

Import, Load, and Review

The steps are the same for importing files, loading the schema, and reviewing dashboards for all versions of Incorta.

A tenant Super User can import the schema, business schema, and dashboard for a given tenant. Next, the tenant Super User performs a full load of the schema. In addition, the Super User can share with various groups and users the dashboards or assign the required access rights to view the dashboards.

Import the Inspector Tool files

A tenant Super User can import the schema, business schema, and dashboards for a given tenant. After successfully downloading the zip files, a tenant Super User imports the schema, business schema, and dashboards.

Import the Inspector Tool schema

Here are the steps to import the Inspector Tool schema for a given tenant:

• Sign in to the Incorta tenant as the Super User.
• In the Navigation bar, select Schema
• In the Action bar, select + New.
• In the Add New Menu, select Import Schema.
• Drag and drop the schema.zip file to the Import Schema dialog.
• In the Import Results dialog, verify two schemas names, _InspectorMetadata and _IncortaMetadata, and select Close.

The _IncortaMetadata schema contains 60 tables and 84 joins.

The InspectorMetadata schema contains the following tables:

• BUSINESS_SCHEMA_VIEWS
• JOINS_DETAILS
• LINEAGE_REPORT
• minifiedLineageReport
• MV_REFERENCED_TABLES
• SCHEMA_TABLES
• VALIDATION

Import the Inspector Tool business schema

Here are the steps to import the Inspector Tool business schema for a given tenant:

• Sign in to the Incorta tenant as the Super User.
• In the Navigation bar, select Business Schema.
• In the Action bar, select + New.
• In the Add New Menu, select Import Business Schema.
• Drag and drop the business_schema.zip file to the Import Business Schema dialog.
• In the Import Results dialog, verify the schema name, _incorta, and select Close.

The _incorta business schema contains the following folder and views:

• IncortaLineage (folder)
  • Inspector
  • Audit_View
• IncortaInspector(folder)
  • validation
  • DashboardLineage
  • Joins
  • MVs
  • BusinessSchemas
  • Schemas
• Metadata (folder)
  • user
  • job
  • schema_job
  • tenant
  • SQL_audit
  • dashboard
  • object
  • group
  • role
  • permission
  • audit
  • action
  • job_old
  • date
  • Hierarchy_Example
  • compaction_job

Import the Incorta Tool dashboards

Here are the steps to import the Inspector Tool dashboards for a given tenant:

• Sign in to the Incorta tenant as the Super User.
• In the Navigation bar, select Content.
• In the Action bar, select + New.
• In the Add New Menu, select Import Folder/Dashboard.
• Drag and drop the dashboards.zip file to the Import Folder/Dashboard dialog.

In the Incorta Tool folder, contains the following subfolders and dashboards:

• Inspector (folder)
  • 0- Run Status
  • 1- Validation UseCAses
  • 2- Data Lineage
  • 2- Unused Entities
  • 2- Impact Analysis
  • 3- Schema Details
  • 4- Audit Summary
  • 5- Tables Used in Business Views
  • 6- Tables Used In Materialized Views
• Metadata (folder)
  • 0.Environment - Tenant Overview
  • 1. Dashboard - User Analytics (BI Audit)
  • 2. Job - Load Summary (Schemas-Tables)
  • 3. Job Execution Analytics (within one day)
  • 4. Dashboards
  • 5. Objects Shared with Users
  • 6. Security
  • 7. User Actions
  • 8. Schema-Table Metrics
  • 9. SQL Query-User Analytics (BI SQLi_Audit)

Load the InspectorMetadata and IncortaMetadata schemas

• Sign in to the Incorta tenant as the Super User or other users that belong to a group with the Schema Manager role and edit access rights to the InspectorMetadata schema.
• In the Navigation bar, select Schema.
• In the Schema Manager, select the _InspectorMetadata schema.
• In the Schema Designer, in the Action bar, select Load → Full Load.
• Repeat the previous two steps for the _IncortaMetadata schema.

Review the Incorta Tools dashboards

• Sign in to the Incorta tenant as the Super User or other users that belong has view access rights to the Incorta Tools folder in the Content Manager.
• In the Navigation bar, select Content.
• In the Card view or List view, select the Incorta Tools folder, and then the Inspector dashboard folder.
• Review the 0- Run status dashboard and confirm the Runtime for the tenant validation.
• Review the 1- Validation UseCases dashboard.
• In the Error & Warning Codes insight, identify all the Severity 1 issues.

You can review additional dashboards. To learn more, review About the Inspector Tool dashboards.

2024.x Inspector Tool enhancements

Starting 2024.1.5, the Inspector Tool generates three new reports as part of its output. You can modify the InspectorMetadata schema and create new tables based on the new CSV files.

• schemaClusters
• tablesDataSources
• mvsProperties

Additionally, the busSchemaViews report has been updated to include business schemas with Analyzer Views only.

schemaClusters report

The schemaClusters report organizes physical schemas within a tenant into clusters based on their interrelationships. Related physical schemas are grouped into distinct clusters while independent schemas are placed in their dedicated cluster: single schema cluster. Schemas are related if they have cross-schema joins or if a materialized view (MV) in one schema references objects in another. This report provides a deeper insight into schema relationships, enabling Schema Managers to create more efficient load plans tailored for optimal performance.

tablesDataSources report

The tablesDataSources report details the physical schema tables within a tenant along with their data sources, allowing Schema Managers to track where each data source is referenced.

mvsProperties report

The mvsProperties report captures materialized views (MVs) in a tenant and lists the additional properties per MV, including Spark and Notebook properties