What's New in Incorta Cloud 2023.4.0

Release Highlights

In this release, Incorta is introducing the dashboard free form layout, as well as major changes in the dashboard manager’s user experience. This release provides improved resilience for the Loader Service, resulting in a huge enhancement in memory handling during load jobs.

Incorta is also introducing the query timeout feature that’s available through a configuration in the Cluster Management Console (CMC). You can also now disable users through the Security Manager.

In addition, Incorta is introducing other new features that aim to enhance Incorta’s performance, decrease exceptions, and empower analytics capabilities.

Upgrade considerations

SQLi queries with user-defined joins or conditions

Incorta has implemented an enhancement to address SQLi queries involving user-defined or unresolved joins within Incorta Engine. In scenarios where the Engine cannot identify a suitable query plan, it falls back to Spark and executes the original user-defined query. However, it's important to note that as a result of these improvements, the outcomes of existing queries may vary compared to previous releases. To maintain consistency with previous results, it is recommended to review these queries accordingly.

API authentication

Starting with 2023.4.0, Incorta introduces a new public API version (v2) that you can use with one of two new authentication methods: personal access tokens (PATs) or Java Web Tokens (JWTs) generated by an OAuth 2.0 authentication server. Public API v1 isn't compatible with the new authentication methods. You can't use the 2023.4.0 release to generate new API keys or renew existing ones that are used in Public API v1. However, you can refresh existing access tokens, or you can use the latest API key that you generated before upgrading to 2023.4.0 and use it to create new access tokens for Public API v1 as usual. If you don't have an API key generated before the upgrade, you can use only Public API v2.

New Features

• Dashboard free form layout
• Enhancing memory handling during load jobs
• Individual and dashboard filter optimization
• Query timeout
• Data lineage enhancements
• A new API version with enhanced authentication
• New API endpoints
• Deactivate Users
• Support abbreviating measure values
• Business Schema Designer enhancements
• Replacing the Load Job Viewer
• Displaying the load job summary
• Enabling Incorta Engine to read from compacted parquet files
• Solving the discrepancy between compacted and committed versions
• Faster formula calculations
• Tooltip tray in components
• Background conditional formatting for KPI components 
• Support Aggregate filter in Pivot table components
• Allow Cloud Diskspace Monitoring configuration

Dashboard free form layout

With this release, Incorta is introducing a new way of creating and managing dashboards. It is also enhancing the user experience when creating or editing a new dashboard.

With the new enhanced dashboards, it is now easier to create, edit, delete, copy, and re-organize your insights using one of the two available layouts that you can choose from the Appearance tab:

1. Grid layout (default)
2. Free-form layout (new)

In the dashboards manager page when you start adding a new insight, the left pane shows two tabs: Add Insight and Appearance.

1. Add Insight – contains all available components including the components you have previously downloaded from the Component SDK marketplace.
2. Appearance – contains the layout types that you can choose from.
3. Layers – Only available when you select free form as your layout, it contains the added insights that you can reorder for layering purposes.

For more information refer to Dashboard Manager.

Grid layout example

Free-form layout example

Enhancing memory handling during load jobs

This release provides improved resilience for the Loader Service, which enhances memory handling during load jobs. The out-of-memory (OOM) exceptions will be reduced significantly by pausing the memory-bound tasks during a load job if the off-heap memory reaches a defined threshold and applying a retrial mechanism if a memory-bound task fails because of an OOM exception.

Individual and dashboard filter optimization

You can now enable Individual and dashboard filter optimization in the Tenant Configurations > Tuning to improve the performance of filtering data, and accordingly reduce the overall rendering time of insights and dashboards.

This feature optimizes filtering data using insight individual filters and when applying runtime security filters and dashboard filters (including dashboard runtime filters, applied filters, dashboard prompts, and filter options).

This feature is disabled by default. To enable it, contact Incorta Support.

Query timeout

New configurations are now available in the Cluster Management Console (CMC), Tenant Configurations > Tuning > Query timeout and Query timeout (in minutes), to set a time limit after which running queries will be terminated automatically, releasing the read lock acquired on the related entities and freeing the resources reserved by these queries.

This option affects querying data in the following scenarios:

• Creating an insight
• Rendering a dashboard
• Sending a dashboard via email or to a data destination
• Downloading a dashboard as CSV or Excel
• Previewing a chart as a listing or aggregated table
• Creating or updating an Incorta Analyzer table or view
• Exploring the data of a table or view
• Creating a data notification condition

The system tracks queries that are automatically terminated, and it records them in the system logs. In addition, a message will be displayed if applicable.

Data lineage enhancements

You can now find out whether your columns are referenced in an insight or in the dashboard filters or prompts.

The Data Lineage Viewer shows both dependent insights and dashboards under the Dependent Dashboards. In the Diagram view:

• Only the dashboard name appears when the column is referenced in the dashboard filters and prompts.
• The dashboard name, tab name, and insight name (if any) appear when the column is referenced in an insight, whether as a column or filter.

In the Tabular view:

• The Dependency Type field shows the specific reference type: Insight or Dashboard.
• The Dashboard Name field shows only the dashboard name when the column is referenced in the dashboard filters and prompts; while it shows the dashboard name, tab name, and insight name (if any) when the column is referenced in an insight, whether as a column or filter.

A new API version with enhanced authentication

The 2023.4.0 release introduces a new version of the Incorta public API (v2) for seamless integration.

• This version doesn’t include endpoints to create or refresh access tokens. You can use your profile drawer to create personal access tokens (PATs) that you can use directly for API authentication. PATs are compatible with Incorta Public API v2 only. Active access keys created before upgrading to 2023.4.0 work only with Public API v1.
• Optionally, you can use OAuth 2.0 as your authenticator for public APIs, knowing that OAuth 2.0 works with Public API v2 only.
• Each user who has Public API access enabled can create and maintain up to 10 PATs.
• Each PAT can have an expiration date.
• You can revoke PATs.
• You can't use a revoked or expired PAT when using the Public API v2 endpoints.

For more information, refer to References → Public API v2.

New API endpoints

The following are the new public API endpoints in 2023.4.0:

• Get SQL connection (/configs/sqlConnection): Gets the connection string and port number of the Incorta SQL Interface (SQLi).
• Query a business view (/query): Queries a single business view into a listing table, aggregated table, or Pivot table.
• List schemas (/schema/list): Returns the list of physical and business schemas that the user owns or has access rights to.
• List schema objects (/schema/{schemaName}/list): Returns the tables, views, and included columns in a specific physical or business schema that the user owns or has access rights to.

Deactivate Users

Incorta administrators (Super Role / User Manager) can now deactivate Incorta analytics users instead of totally deleting them. Deactivating users will not delete their data, instead it will keep the data saved and usable for others. Also, deactivated users’ scheduled jobs will remain active unless the administrator chooses to suspend them.

To deactivate a user or multiple users:

1. Log in to Incorta as a SuperRole / User Manager.
2. Select Security.
3. Select one or more users.
4. In the Search bar, select More Options (︙vertical ellipsis icon).
5. Select Deactivate User(s).
6. Confirm the users you need to deactivate.
   

To activate a user or multiple users:

1. Log in to Incorta as a Super Role / User Manager.
2. Select Security.
3. Select one or more users.
4. In the Search bar, select More Options (︙vertical ellipsis icon).
5. Select Activate User(s).

Upon deactivation, the user cannot log in to Incorta. If they try, they will encounter a wrong username or password error. When you activate a user, they will receive an email with a URL allowing them to log in to Incorta.

Support abbreviating measure values

In this release, Incorta is supporting abbreviating measures and enabling you to configure how the abbreviation behaves.

The Abbreviation field has the following possible values:

When creating a new insight or migrating one, the default value for the Abbreviation field is Auto. For KPI insights, it’s different:

1. When creating a new KPI, the default Abbreviation for measure settings is Inherit and the default Abbreviation in general settings is Auto.
2. Upon upgrade, existing KPI insights will have the default Abbreviation in measure settings set to Inherit and the default Abbreviation in general settings set to None.

Business Schema Designer enhancements

This release enhances the Business Schema Designer by introducing a new user interface (UI) and improved features. The new UI enriches the user experience and facilitates managing views in a business schema. The new UI has the same look and feel as other components, such as the Analyzer and the Catalog (Content Manager), which provides a consistent user experience.

Two modes are available when dealing with a business schema: View mode and Edit mode. A user that owns or has Edit access rights to the business schema can access both modes while a user with View or Share access rights can access only the View mode.

In the Edit mode, you can add, edit, and delete views and folders. You now have the option to remove multiple columns from a view at once. If you try to exit the Business Schema Designer while there are unsaved changes, a warning message appears, allowing you to save these changes. You can save the changes to the current version of the business schema or discard them. Once you save or discard the changes, you are back to the View mode.

In the View mode, you can do the following according to your access rights:

• Create a copy of the business schema
• Delete it
• Edit its details
• Explore its data
• View and manage its share options
• View, export, and restore its versions
• Access the Edit mode

Unlike previous releases, you can now add, edit, and delete business schemas when the Loader Service isn’t available or running.

Replacing the Load Job Viewer

The Load Job Details Viewer introduced in 2022.12.0 replaces the Load Job Viewer starting this release. Now, when you select the status or time of the latest or current load job on the Schema Manager → Schemas or the Schema Designer or you select Job Details of a failed, interrupted, or finished with errors job on the Home page, the job opens in the Load Job Details Viewer.

You can use the Load Job Details Viewer to check the details of load jobs and load plans triggered for a specific physical schema, regardless of how the job was started: manually through the application, by a scheduled load plan, or via an API endpoint.

Displaying the load job summary

The Load Job Details Viewer now shows the summary of the selected load job.

The summary part shows the following details:

• The total duration of the load job (introduced both visually and numerically)
• The duration of each stage in the job
• The number of schemas in the load job
• The number of tables (physical tables, MVs, and derived tables) in all physical schemas

Enabling Incorta Engine to read from compacted parquet files

Incorta Engine can now fully read columns from compacted parquet files if the compacted version is consistent with the committed version. However, it will read columns from the source parquet files when reading columns incrementally or when there is a discrepancy between the compacted version and the committed version. This feature unifies the source that both Incorta Engine and Spark read from. It also decreases the I/O operations and enhances the performance of reading columns when a table has too many duplicates.

Solving the discrepancy between compacted and committed versions

To ensure that Incorta Engine and Spark have access to the same version of data at a point in time, the Loader Service will make sure that both the compacted and committed (loaded) files of a physical schema object share the same version. The Loader Service synchronously integrates the compaction step of performance-optimized tables in the load job process. If the compaction of an object fails, loading the object fails as well.

Faster formula calculations

This release improves the parallel formula materialization process to ensure faster calculations of grouping dimension formula columns that reference string columns with a large number of records. This enhancement affects dashboard insights, Analyzer tables, Analyzer views, and data notification conditions.

Tooltip tray in components

The tooltip tray is now available in more components. Having the tooltip tray enables you to add whatever information you need to show when hovering on your insight.

The supported components in this release are:

• Sunburst
• Combination
• Time series
• Line time series
• Area range
• Stacked column and line
• Dual x-axis
• Waterfall
• Bubble map
• Funnel
• Pyramid
• Packed bubble
• Organizational

For more information, refer the corresponding document for each visualization.

Background conditional formatting for KPI components 

Background conditional formatting is now available for KPI components. You can apply the conditional formats to a measure column in a KPI component for both font color and background.

When you add a measure to the Measure tray, you can set the conditional formatting for the background through a new option Background. You can choose from the already existing colors or define a custom color from the color palette.

For more information, refer to Visualizations → KPI. 

Support Aggregate filter in Pivot table components

Incorta now supports aggregate filters for Pivot table components. In this release, you can find the Aggregate Filter tray added to Pivot table components. You can now add measure to the aggregate filter tray, which will reflect on the data you are trying to show.

For more information, refer to Visualizations → Pivot Table.

Allow Cloud Diskspace Monitoring configuration 

Incorta is adding a new configuration to the CMC. The Allow Cloud Diskspace Monitoring option is under Clusters > Cluster Configurations > Server Configurations > Diagnostics. When you enable this option, you will be able to see the Shared Directory Diskspace under Monitoring > Tenants. This configuration is applied to cloud storage only and it is disabled by default. Enabling this configuration is not recommended and might increase the number of I/O operations on your cloud storage. For more information, refer to Guides → Configure Server.