Release Notes 6.0

Release Highlights

The 6.0 on-prem release is introducing multiple major features that will significantly impact your experience in multiple Incorta modules.

In this release, you can find the dashboard free-form layout, which enables you to create and manage dashboards with ease. Also, Incorta is introducing the Component SDK that enables you to develop your own visual components and use them in dashboards. In addition, you can download pre-developed components from our external marketplace.

In addition to the previous features, Incorta is having a new improved loader that has optimized load job calculations and support multi-schema load jobs, also a new enhanced data delivery capabilities by automating many configuration steps and adding Google BigQuery as a data destination.

SQLi is now a separate service that you can add to your node. Incorta is also introducing the graphical data lineage, which enables you to track column dependencies across different entities throughout Incorta.

In addition, you can also find multiple new public API endpoints and enhanced the product security by upgrading Spark to version 3.3 and jQuery to version 3.6.0.

Upgrade Considerations

Support of metadata database

The 6.0 release has MySQL as a metadata database. Oracle metadata to be available in subsequent releases.

Important

This release uses the Data Agent version 6.1.0. Please make sure to upgrade to this data agent version.
User-defined load order may result in load job execution plans with cyclic dependencies, leading to load job failures. To avoid such failures, it is recommended to delete MVs' user-defined load order before upgrading to 6.0.
After running the Apache Parquet Merge tool in this release, you must load the respective physical schemas from staging.
In the case of using external Notebooks, you must force reinstalling the Python library after upgrading to 6.0.
Ask your CMC admin to configure your SQLi service in the CMC if you are using a custom installation.

Spark 3.3 upgrade

Incorta upgraded to Spark 3.3 in the 6.0 release. Before you start upgrading to this release, make sure to run the Spark compatibility check tool spark-upgrade-issues-detector.sh located in the 6.0 extracted folder, and to have the compatible libraries as mentioned in the Support for Spark 3.3 section.

To run the tool, use the following command:

sh <EXTRACTION_DIR>/spark-upgrade-issues-detector.sh

The tool will create two files:

potential-upgrade-issues.csv, which contains the potential compatibility issues that you have.
spark-issues-references.xlsx, which contains examples and references to the compatibility concerns and how can you fix them.

Make sure to handle the compatibility concerns reported in the .csv file, before your proceed to upgrade.

Copying core-site.xml file in Azure

Incorta has enhanced the upgrade process in case you are using cloud file system. You must backup your core-site.xml file and apply the following steps during the upgrade before starting your cluster.

Important

Do not paste the file manually.

Place the backup core-site.xml under cmc/bin and IncortaNode/bin.
Run the following locations script files, using the python command (for example, python3 update_core_site_incortaNode.py):
- <INCORTA_HOME>/cmc/bin/update_core_site_cmc.py
- <INCORTA_HOME>/IncortaNode/bin/update_core_site_incortaNode.py

Incorta data delivery

With the new enhanced Incorta data delivery (previously known as Incorta Intelligent Ingest) you must be aware of the following before the upgrade to this release:

The used python package is replaced with an Incorta Java-based component.
No Wheel or EGG file required.
No configurations under the spark-defaults.conf needed.
Place synapse_mappings.csv at <IncortaNode>/Synapse directory.
You must perform full ingest.
If the schema has target schema name or a table prefix used, need to rename the setting per schema.
You can adjust the default string length per schema, knowing that the default value is 4000.
Open the data destination definition after the upgrade and set the right authentication option in the core-site.xml file based on the Incorta tenant directory configurations.

Schemas with user-defined load order

With the introduction of the next-generation loader, Incorta automatically detects inter-object dependencies within a load plan during the Planning phase of a load job. The Loader Service utilizes these detected dependencies and the user-defined load order within the schema to create an execution plan for loading objects. However, it’s important to note that using both the MVs' user-defined load order and automatically detected dependencies may result in an execution plan with cyclic dependencies, leading to load job failures. To avoid such failures, it is recommended to delete the MVs' user-defined load order before upgrading to the 6.0 release.

Password policy

The default password policy is changing starting with 6.0. The default policy is applied only to newly created passwords, whether for new users creating their first passwords or existing users changing their passwords. However, this will not require existing users to change their passwords. The new password policy mandates the following:

The password length must be between 12 and 32 alphanumeric characters.
A password must contain at least one uppercase letter, one lowercase letter, one special character, and one digit.
A password cannot include the user login name or email address.

daysBetween function

The daysBetween function now returns the value as double. To round the returned value and retain the old behavior, you must wrap the daysBetween function with the round or ceil functions.

It is recommended to load the physical schemas that might be using the daysBetween function from staging.

MVs with explicit decimal casting

After upgrading to this release, MVs with columns that are casted to decimal in the MV script fail to validate or load with a mismatched types error. To avoid this failure, do one of the following:

Change the existing cast function to match the MV column data type in the Table Editor.
Change the MV column data type to decimal in the Table Editor and revalidate the MV definition. This method will affect downstream dependencies like formulas, functions, or insights referencing the changes columns. For more information, refer to References → Decimal Data Support.

SQLi Queries with user-defined joins or conditions

Incorta has implemented an enhancement to address SQLi queries involving user-defined or unresolved joins within Incorta Engine. When the Engine cannot identify a suitable query plan, it falls back to Spark and executes the original user-defined query. However, it's important to note that because of these improvements, the outcome of existing queries may differ compared to previous releases. To maintain consistency with previous results, it is recommended to review these queries accordingly.

API authentication

Starting with 6.0, Incorta introduces a new public API version (v2) that you can use with one of two new authentication methods: personal access tokens (PATs) or Java Web Tokens (JWTs) generated by an OAuth 2.0 authentication server. Public API v1 isn't compatible with the new authentication methods. You can't use the 6.0 release to generate new API keys or renew existing ones that are used in Public API v1. However, you can refresh existing access tokens, or you can use the latest API key that you generated before upgrading to 6.0 and use it to create new access tokens for Public API v1 as usual. If you don't have an API key generated before the upgrade, you can use only Public API v2.

New Features

Dashboards, Visualizations, and Analytics

Dashboard free form layout

With this release, Incorta is introducing a new way of creating and managing dashboards. There are also enhancements to the user experience when creating or editing a new dashboard. With this change, you have more flexibility and improved ease of use when designing dashboards. For more information, refer to Dashboard Manager.

Flexibly arrange and resize insights in dashboards

To make your dashboards more flexible and personalized, simply use the Personalize feature to rearrange and resize insights within the grid-layout dashboard while editing its layout.

You can stack charts on each other in the same row and make insights that span multiple rows. You can also freely resize the insights in the following ways:

For multiple insights within the same row, drag and drop the horizontal border line to resize all insights in that row.
For a single insight, drag and drop the vertical border line to resize that insight.
Drag and drop an insight below another one to coincide with a third within the same row.

For more information, refer to Guide → Analyze.

Incorta Component Software Development Kit

In this release, Incorta introduces the Incorta Component SDK, enabling customers and partners to create custom visual and non-visual components for integration into Incorta Analyzer and Dashboards.

Important

Using this new feature requires React knowledge as you will be developing HTML, CSS, JSON, Javascript, Node.js, NPM, React, and Typescript.

The Component SDK is enhanced with an API called PrivateData. This API enables developers to store any data that is not derived from configurations, settings, or measures. You can use this API to store any data while rendering the developed component.

The Component SDK comes with an external marketplace where you can download any component you want. As an admin user, you can upload the downloaded component or your own developed component using the External Components section in the Insights panel.

One of many useful external components available in the marketplace, is the Slicer. The Slicer is a list or range selector created in the Analyzer.

For more details and information about the Incorta Component SDK, how to use it, refer to References → Incorta Component SDK.

Copy dashboard link

Incorta is introducing a new method to share a dashboard with other users. Now you can share a dashboard link via copy and paste.

You can find the Copy link option in the following places:

The More options (⋮ vertical ellipsis) icon in the Content manager page beside a dashboard.
Share icon in a dashboard's Action bar.
Dashboard's tab More options (⋮ vertical ellipsis) icon.

In sharing the dashboard link, dashboards filters and bookmarks by default. By selecting "Include current filters in the URL" option, the filters will be added to the URL string. This can be especially useful when leveraging URLs from third-party tools.

Note

Use the Include current filters in the URL option if you are drilling down to Incorta or embedding the dashboard.

Conditional formatting for column and bar visualizations

Conditional formatting is now available for Column and Bar visualizations. You can apply multiple conditional formats to a measure column in a bar or a column chart insight that has only one measure pill and no coloring dimension pills. Conditional formats are preserved when changing a tabular insight to a bar or column insight and vice versa.

Conditional formatting based on another measure

For chart and table insights that support conditional formatting (Bar, Column, KPI, Listing Table, Aggregated Table, and Pivot Table insights), you can now apply different formats based on a measure (column or formula) depending on the value or the aggregated value of another column or a formula. You can add multiple conditional formatting rules to the same measure and arrange them according to the order of application.

KPI visualizations enhancements

The KPI visualization adds two new key functionalities:

Grouping: The KPI insight supports multiple tiles, one for each dimension value.
Support for prompt filters
Drill-down support
Conditional formatting

In addition, you can now format the style and appearance of KPIs in dashboards from the insight Settings. This includes the following:

Font size, family, and color for the KPI measure value
Alignment of values and labels (left, center, or right)
Background color for the KPI tiles
Gutter size, or margin between KPI tiles, in pixels
Space between labels and values in pixels

For more information, refer to Visualizations → KPI.

New Tornado visualization

The new Tornado chart is a new visualization type available in Incorta. The Tornado visualization is used for sensitivity analysis by comparing one variable with the output of another independent variable.

For more information, refer to Visualizations → Tornado.

Excel and CSV enhancements

The performance of exporting Excel and CSV files has improved. Excel and CSV file exports now reflect the date formats displayed in Incorta Analytics.

Enhanced the SQL generation link

The function of the SQL link is enhanced to view the generated SQL reference code from within an insight for aggregated tables, listing tables, and pivot tables.

Incorta added a new tray, Tooltip, in the Analyzer for several visualizations. You can add different types of pills to this tray, such as number, string, and date; it will appear as a tooltip when you hover over a certain point at the insight. Adding pills to this tray is optional.

Note

Incorta treats all column types added to the tooltip tray as measures.

The tooltip tray is currently available for the following visualizations:


Stacked Line	Organizational
Bar	Packed bubble
Percent Bar	Pyramid
Stacked Bar	Funnel
Column	Bubble map
Percent Column	Waterfall
Stacked Column	Time series
Area	Line time series
Percent Area	Area range
Stacked Area	Stacked column and line
Bubble	Dual x-axis
Donut	Scatter
Pie

Dashboard runtime filters enhancements

This release enhances the user experience and usability of displaying and updating dashboard runtime filters. When selecting a pill on the filter bar, the filter details appear just below the filter pill on the filter bar rather than the Filter dialog. This allows updating the filter details without moving to the top of the page.

Toggle prompt filters for individual insights on a dashboard

Using the new Inherit Runtime Filters toggle located in each insight's settings, you can individually manage which insights your prompt filters affect. Currently, the majority of insights, including all table insights, support this feature.

Individual and dashboard filter optimization

You can now enable Individual and dashboard filter optimization in Tenant Configurations > Tuning to improve the performance of filtering data, and accordingly reducing the overall rendering time of insights and dashboards. This feature optimizes filtering data using insight individual filters and when applying runtime security filters and dashboard filters (including dashboard runtime filters, applied filters, dashboard prompts, and filter options).

Note

This feature doesn't affect queries with non-regular expressions, such as First Version, Last Version, and Distinct.

Hide presentation variables from filter bar

This release is introducing a new option that enables you to hide presentation variables from your dashboard filter bar using the Hide from filter bar option. You can enable this option while setting up the presentation variable in the Dashboard Filters Manager.

Important

Regardless of enabling this option or not, Incorta will apply the default value of the presentation variable to your dashboard.

For more information refer to Concepts > Presentation Variable and Tools > Dashboard Filters Manager.

Enhanced Between filter for dates

Now you can filter your insights in a dashboard using the between operator based on dates. Incorta now enables you to use System Variables to specify a date range by typing in the dollar sign $ in the search field.

Important

You must enter the dollar sign $ so you can list the available system variables.

The Tag Cloud visualization now has the Color By tray to enable you to use coloring as an additional dimension to view your data.

For more information, refer to Visualizations → Tag Cloud.

Dashboard delivery support for bookmarks

You can now create scheduled jobs to deliver a dashboard as PDF or HTML to export folders or via email while applying a selected bookmark. You can also choose to reveal the bookmark name and the values of the applied filters.

For more information, refer to Guides → Analyze → Manage Dashboard Bookmarks.

Bubble chart size enhancements

You can now set minimum and maximum percentage values for the radii of the bubbles within a bubble visualization. You can do that using the newly added property Radius in the pill properties of Size By tray.

Support abbreviating measure values

In this release, Incorta supports abbreviating measures and enables you to configure how the abbreviations behave.

The Abbreviation field has the following possible values:

Value	Description
None	Displays the measure as calculated with no change.
Auto	Default. It detects the measure value and abbreviates accordingly.
Thousands	Abbreviates the measure value to thousands.
Millions	Abbreviates the measure value to millions. For example, `3,000,000` → `3M`.
Billions	Abbreviates the measure value to billions. For example, `3,000,000,000` → `1B`.
Trillions	Abbreviates the measure value to trillions. For example, `3,000,000,000` → `0.003T`.
Inherit	Available in KPI visualizations only. It inherits the Abbreviation value from the insight's General Settings.

Note

The abbreviation field is not available for Table visualization.

When creating a new insight or migrating one, the default value for the Abbreviation field is Auto. For KPI insights, it's different:

When creating a new KPI, the default Abbreviation for measure settings is Inherit and the default Abbreviation in general settings is Auto.
Upon upgrade, existing KPI insights will have the default Abbreviation in measure settings set to Inherit and the default Abbreviation in general settings set to None.

Bursting reports

Incorta now supports security context-based bursting reports, where reports are distributed based on the recipients' context instead of the sender's context. For example, if salespeople in different regions each need a report showing the sales target for their country or region, use bursting reports to send each salesperson only the information they need.

To burst a report, in the dashboard Send/Schedule a Report page, select the Burst Report check box. You can send or schedule a bursting report or dashboard for up to 300 internal users with Incorta accounts and at least view access rights to the dashboard. The report runs once but renders customized data based on the security filters or session variables of each recipient.

Note

Bursting reports is an Incorta Labs feature in this release. To enable it on your cluster, turn the Enable Bursting Reports option on using Cluster Management Console (CMC) > Tenant Configurations > Incorta Labs.

Rendering and downloading insights performance enhancement

Queueing of download and render requests: The Analytics Service now queues requests for the same insight with the same format, preventing simultaneous execution of duplicate requests.
In-memory caching of downloaded insights: the Cluster Management Console (CMC) admin can configure the Analytics Service to maintain an in-memory cached version of insights downloaded, sent to data destinations, saved in a download folder, or shared via email in XLSX and CSV formats.

The Analytics Service checks for a cached version before executing the insight query, regardless of the requesting user. The new Export to CSV/XLSX caching limit (In megabytes) option, located in CMC > Tenant Configurations > Tuning , controls the maximum data size of the query result that the Analytics Service caches for downloaded insights. By default, this feature is disabled. To enable it, set the caching limit to a value greater than 0. Insights with a query result size exceeding the defined limit will not be cached. Additionally, if any of the Maximum Cached Memory (%) or Maximum Cached Entries options are disabled or exceeded, the Analytics Service won't cache downloaded insights.

These improvements collectively enhance performance, optimize resource utilization, and improve the overall user experience within the Analytics Service.

Data Management Layer

New Native PostgreSQL Parser

Incorta introduces a new parser that is based on the native PostgreSQL library version 13. The new parser aims to support more PostgreSQL queries and functions, which will enhance the experience of the SQL Interface (SQLi) users, as well as the integration with external visualizations tools, such as Tableau and Microsoft Power BI. It also enhances query execution performance.

Important

The new parser accepts only valid PostgreSQL syntax.

Expanded Data Delivery capabilities

Incorta now has Google BigQuery and Microsoft Azure Synapse Analytics destinations for data delivery. Our data destinations feature allows for the ingestion and enrichment of data and the transfer of full and incremental loads to other cloud analytics platforms. For more information, refer to the Incorta Data Delivery configuration documents.

SQLi as a standalone service

Starting from this release, Incorta provides the SQLi as a separate service on CMC. You can connect various BI tools to SQLi. Add the SQLi service to the node you are using:

Navigate to your cluster
Select the Nodes tab
Select More Options (⋮ vertical ellipsis icon)
Select Add Service > SQLi

You must set the Cache size setting under Cluster Configurations > SQL Interface of your cluster to be 25% or less than the allocated memory for the SQLi service.

SQLi service management includes the following options, create, edit, delete, start, stop the service, show and download SQLi logs, and monitor the memory usage of the service.

Important

If you are using custom installation, you must configure your SQLi service in the CMC.

Known issues

Issue	Workaround
Using internal session variables with SQLi service might cause out of memory error.	Add `-Dengine.max_off_heap_memory=<value in Bytes>` in the `<installationPath>/IncortaNode/sqli_extra_jvm.properties`, where the SQLi service is installed to set an off heap limit. For example, `-Dengine.max_off_heap_memory=10737418240` will set the off heap memory to 10 GBs.

Global Variable Enhancements

This release introduces multiple enhancements to global variables regarding definition and support. You can define global variables using the query and queryDistinct functions in the Formula Builder instead of specifying a constant value. In addition, you can reference global variables in new contexts, such as formula columns and other global variables.

After upgrading to 6.0, your pre-existing global variables will continue to execute as expected. However, if you attempt to edit them, you must make sure to adhere to the new syntax by using query or queryDistinct. For example:

Existing value	Data Type	New Syntax
`1234`	integer, double, or long	`query(1234)`
`1234`	string	`query('1234')`
`Name`	string	`query('Name')`
`2022-01-03`	string	`query('2022-01-03')`
`2022-01-03`	date	`query(date('2022-01-03'))`

Referencing a global variable in one of the new contexts (usually in the Formula Builder) is different from referencing it in the extraction queries or scripts of physical schema tables and MVs. In the Formula Builder contexts, precede the global variable name with a single dollar sign $, rather than a double dollar sign $$. The data type of a global variable value is no longer manually selected; it is determined by the query syntax and referenced columns.

For more information, refer to Concepts → Global Variable.

Business Schema enhancements

This release enhances the Business Schema Designer and Business Schema Manager significantly by introducing a new user interface (UI) and improved features. The new UI enriches the user experience and facilitates managing views in a business schema. The new UI has the same look and feel as other components, such as the Analyzer and the Catalog (Content Manager), which provides a consistent user experience.

Two modes are available when dealing with a business schema: View mode and Edit mode. In addition, you can now add, edit, and delete business schemas even when the Loader Service isn't available.

The enhancements to the Business Schema Manager and Business Schema Designer include keeping track of business schema versions and introducing additional columns and classifications of business schemas. Incorta now maintains business schema versions for better collaboration, business schema lifecycle management, and audit capabilities. The Business Schema Manager also has columns in the business schema list, such as the Department , Modified by , and Last Modified columns. You can specify the displayed columns and sort the business schema list using the new columns.

In addition, as the Super User or a schema manager, you can make copies of business schemas that you own or have Edit access rights to. The new copy inherits the description and department of the original business schema; however, you can edit these configurations in addition to the business schema name. You can create a copy of only one business schema at a time.

Data Lineage graphical representation

This release introduces a new tool, Data Lineage Viewer, so that you can track the dependencies of each column (data columns or formula columns) in your physical schema and business schema objects. Dependent entities include physical schemas, business schemas, and dashboards. You can preview a list of entities where a column is referenced, whether directly or via another dependent entity. You can preview the dependency list in a tabular or graphical view. You can also download this list to a csv file.

Schema Diagram Viewer improvements

Incorta is introducing a new improved user interface (UI) and enhanced functionality in the Schema Diagram Viewer, such as better usability and visibility when viewing and managing cross-schema join relationships. External schemas are now collapsed, and each is displayed as a node with all join arrows that relate to the same object in the current schema combined in only one arrow.

You can now use the Schema Diagram Viewer to easily construct, navigate, and maintain the schema data model. You can create, edit, and delete joins directly in the Schema Diagram Viewer. In addition, the details drawer has the color of the object icon indicates the object type, and the icon of a join column denotes the column data type. The diagram legend displays a key for external schemas with their number. A visual indication appears when you hover over the node of an object denoting that you can create a new join relationship.

For more information, refer to Schema Diagram Viewer.

Executing a portion of an SQL script

To facilitate the iterative process of writing SQL in Incorta, now you can test and execute only a portion of the SQL script when creating or updating an Incorta SQL View, Incorta SQL Table, custom SQL table in the Schema Wizard, or SQL-based data source.

Important

For custom SQL tables, make sure to execute the final script you want before saving as the Schema Wizard will use only the portion you execute to create the custom SQL table. For Incorta SQL Views, Incorta SQL Tables, and SQL-based data sources, keep only the script you want for creating each.

Enhanced support for the PostgreSQL INTERVAL function

Incorta enhanced the support for the PostgreSQL arithmetic functions. The following is a list of all the newly added enhancements for the INTERVAL function:

Double in INTERVAL string, for example, INTERVAL '1.5 years'
PG_VERBOSE, SQL_STANDARD, ISO8601, and ISO_ALTERNATE
EXTRACT/DATE_PART with INTERVALs, EXTRACT
EXTRACT/DATE_PART with TIMESTAMP
TIMESTAMP/DATE + INTERVAL
TIMESTAMP/DATE - INTERVAL
INTERVAL + INTERVAL
INTERVAL - INTERVAL
INTERVAL * double_values
INTERVAL / double_values
INTERVAL * columns(numeric type)
columns(numeric type) / INTERVAL

Refer to the PostgreSQL documentation for more information and examples.

Additional Considerations

Support for TO_CHAR with INTERVAL types is not yet supported.
CAST(Column as INTERVAL) works for string constants like CAST('1 year' as INTERVAL).
Incorta supports precision up to milliseconds only.

BlackLine data destination

You can push the data in the pre-built templates to a BlackLine data destination via a secure FTP site. You can schedule more frequent data transfers to support month-end close activities and access detailed transaction data, which enables business users to quickly resolve reconciliation issues. Incorta enables you to send encrypted data files to BlackLine data destinations.

Refer to Concepts → Data Destination, for instructions on configuring BlackLine as a data destination.

Enhanced data files upload

Incorta has improved the user interface of uploading files and folders. The user interface is now more informative throughout the entire workflow. Also, you can now upload a folder or multiple files seamlessly. The Local Data Files tab now shows the files/folders uploaded ordered with the last modified files/folders on top.

Oracle WMS Cloud connector

The new Oracle Warehouse Management Software (Oracle WMS Cloud) application connector brings your logistics data into Incorta so you can start investigating your data and building insights. Oracle WMS Cloud is the logistics system in the Oracle Supply Chain Management and Manufacturing system.

AlloyDB Connector

Incorta is introducing a new connector that integrates with the Google-powered AlloyDB. AlloyDB is a PostgreSQL-based distributed database. It combines performance, safety, availability, and scalability features.

Note

The Incorta AlloyDB connector is available for preview only.

For more information, refer to Connectors → AlloyDB.

SAP ERP connector callback extraction

Incorta is introducing a new callback extraction mode in the existing SAP ERP connector. The new extraction mode is faster and easier to set up.

Note

It is recommended to use the callback mode instead of the asynchronous mode for better performance.

For more information refer to Connectors → SAP ERP.

Support for SparkR

Incorta supports SparkR for creating materialized views (MVs). For more details on SparkR, refer to SparkR documentation.

Public APIs

New API version with enhanced authentication

This release introduces multiple enhancements to Incorta's public API. These enhancements include the following:

A new version of Incorta public API (v2) is now available for seamless integration.
Enhanced public API authentication, where you can create and maintain up to 10 personal access tokens (PATs) that you can directly use in the request header for authentication. PATs are compatible with Incorta Public API v2 only. Active access keys created before upgrading to 6.0 work only with Public API v1.
Optionally, you can use OAuth 2.0 as your authenticator for public APIs, knowing that OAuth 2.0 works with Public API v2 only.

For more information, refer to References → Public API v2.

New API endpoints

The following are the newly introduced public API endpoints in 6.0:

Query a business view (/query): Queries a single business view into a listing table, aggregated table, or Pivot table.
List schemas (/schema/list): Returns the list of physical and business schemas that the user owns or has access rights to.
List schema objects (/schema/{schemaName}/list): Returns the tables, views, and included columns in a specific physical or business schema that the user owns or has access rights to.
Load Schema Endpoint (/schema/load): Starts the load of one or more physical schemas.
Schema Status Endpoint (/schema/{schemaName}/status): This endpoint retrieves the details of a specific physical or business schema. Details include the available objects (tables or views), the number of columns per object, creation and modification time, and schema owner. For physical schemas, it returns the status of the most recent model update and load jobs and the data size.
Send Report (send-report): Use this endpoint send the dashboard as a report via email.
Transfer Ownership Endpoint (/users/transferOwnership): This endpoint transfers the ownership of all entities that a user owns to another active user. You can also transfer all the share permissions assigned to this user to the other user.
Reset Password Endpoint (resetPasswords ): As a User Manager or a Super User, you can use this API endpoint to instantly reset the password of all Incorta authenticated users. Alternatively, as an admin, you can enable password expiration so that each Incorta authenticated active user will have to change the password at a scheduled cadence. For more information, refer to Guides → Configure Tenants → Incorta Authentication.

Note

The /schema/list and /schema/{schemaName}/list endpoints are available in Public API v2 only.

Architecture and Application Layer

Next-generation loader

Incorta's next-generation loader comes with a new architecture that comprises multiple components. The new loader executes load jobs against a Directed Acyclic Graph (DAG) and helps to detect dependencies between objects in the same load job.

The new architecture optimizes the post-load calculations so that the Loader Service concurrently executes different calculation types. These enhancements eliminate previous limitations and open the door for more improvements. They also bring several benefits, including the ability to:

Create a load plan for multiple schemas, reducing the total number of scheduled loads
Run table extraction simultaneously with transforming MVs
Optimize post-load calculations
Provide enhanced load job details and a new screen for monitoring recently executed load jobs
Improving the deduplication process

Important

Although the new architecture still allows for multiple Loader Services to handle data acquisition and ingestion within the same cluster, it doesn’t support schema distribution for multi-schema load jobs. Therefore, if you are utilizing multi-schema load jobs, it is important to avoid using schema distribution.

Memory handling enhancements during load jobs

This release provides improved resilience for the loader service. Memory handling is enhanced significantly by pausing memory-bound tasks during a load job if the off-heap memory reaches a defined threshold and applying a retrial mechanism if a memory-bound task fails because of an OOM exception.

Note

As a result of this feature, some load jobs may take more time than expected due to the wait or retry time. The system will track these cases and record them in its logs.

Enabling Incorta Engine to read from compacted parquet files

Incorta Engine can now fully read columns from compacted parquet files where applicable if the compacted version is consistent with the committed version. However, it will read columns from the source parquet files when reading columns incrementally. It decreases I/O operations and enhances the performance of reading columns when a table has many duplicates.

To improve the effectiveness of reading fully from compacted data in both the Loader and the Analytics, the deduplication step (Compaction) is integrated as a synchronous step within the ETL job.

Note

This feature may lead to increasing the overall load job time.

Faster formula calculations

This release improves the parallel formula materialization process to ensure faster calculations of grouping dimension formula columns that reference string columns with many records. This enhancement affects dashboard insights, Analyzer tables, Analyzer views, and data notification conditions.

PK-Index Calculations default behavior change on Full Load

Starting this release, the Primary Key (PK) index calculations during a full load job will be disabled by default for newly created physical tables and MVs with one or more key columns. The new behavior will not affect existing tables and MVs. Enable the Enforce Primary Key Constraint toggle in the Table Editor if the source of your table or MV has duplicate data that you do not want to bring to Incorta. Disabling this option enhances the load time and performance and memory requirements during full load jobs. However, this may cause data redundancy.

MV Assistant: recommended values for better MV performance

The MV Assistant helps you identify the appropriate value combination of Spark-related configurations for each materialized view (MV). A dedicated dashboard displays the recommended values and a comparison between the performance of the current and recommended configurations. In addition, the metadata database saves these recommended values per MV. As a schema developer, you can manually apply the recommended values per MV.

The admin must enable the new feature, define the related properties in the Tenant Configurations, and import the MV Assistant dashboard and underlying schemas.

Note

The MV Assistant is a preview feature in this release.

For more information about the available properties, refer to Tenant Configurations → MV Assistant.

Decimal data types support

Incorta can now discover, read, store, and manipulate decimal data from different data sources, ensuring high-precision calculations. Incorta supports decimal numbers with a precision of up to 18 while it discovers and stores decimal data with a precision higher than 18 as double. Formulas and functions also support the decimal data type.

For information about data sources, functions, and components that support the decimal data type, refer to References → Decimal Data Support.

Handling null values

The Cluster Management Console (CMC) admin can now control how Incorta handles null values using the Server Configurations > Customizations > Null Handling option. This option is a preview feature that is disabled by default. The CMC admin can turn it on so that Incorta handles null values as null rather than real values, such as an empty string or zero. When null values are handled as real values, there is an inconsistency between function and formula results in Spark-based objects (such as materialized views) and Incorta-native objects (such as physical schema tables and Incorta SQL and Analyzer tables). The inconsistency also expands to other areas, such as filters, grouping, and subtotals.

When you enable Null Handling, Incorta differentiates between null values and real values in the following areas:

Aggregation functions: SUM, MIN, MAX, AVERAGE, MEDIAN, COUNT, and DISTINCT
Aggregation formulas using SUM, MIN, MAX, AVERAGE, MEDIAN, COUNT, and DISTINCT
Aggregation formulas using level-based measures
Grouping dimensions based on physical columns
Totals and subtotals

Note

Enabling this feature may result in a different output for existing calculations.

Mandating compaction

Starting this release, the Enable Always Compact option won't be available on the CMC as creating a compacted version of parquet files during load jobs is no longer an option. The Loader Service will start a compaction job during all load jobs. During loading from staging, the Loader Service initiates a compactions job if it detects issues with the compacted version of an object.

Notebook enhancements

Query methods

The Incorta Notebook now gives data engineers and data scientists the ability to author SQL and PostreSQL queries directly inside the Notebook. Previously, users were required to read an entire table into Notebook and write additional code to parse the extracted data.

For more information, refer to Concepts → Materialized view.

Dynamic form functions

The Incorta Notebook now includes dynamic forms, allowing analysts and data scientists to create interactive Python notebooks for demonstrations, presentations, and analysis. You can create statically or dynamically populated form controls, such as a dropdown menu, to quickly change desired fields for analysis. For more information, refer to Tools → Notebook Editor.

Support for fiscal calendars

To empower data engineers in the financial sector, Incorta now natively supports fiscal year calendars. This feature simplifies the setup and consumption of fiscal calendars in Incorta and allows for the rollup of transactional data into fiscal calendars. You can define the start of the fiscal year in the Cluster Management Console (CMC) under Server Configurations > Customization.

Date filters and date aggregations now honor fiscal calendars. In addition, multiple fiscal-calendar-based system variables are now available to select from when creating filters, formula columns, or queries. You can also apply different fiscal-calendar formats depending on the selected Date Part. You can use these new variables and date parts along with the existing ones to create comparisons between fiscal and Gregorian calendars without much effort. The ago and toDate functions now support the fiscal calendar as well.

Security and Integrations

Support for Spark 3.3

Starting with this release, Incorta supports Spark 3.3, which provides numerous enhancements. The 6.0 release is bundled with Spark 3.3, which requires some configuration and environment updates.

Use the Spark compatibility check tool spark-upgrade-issues-detector.sh to spot out configurations and MV scripts that might be incompatible with Spark 3.3. Check the upgrade considerations section.

Prerequisites

Customers who are using external Spark must upgrade it to Spark 3.3.
Customers who are upgrading from previous Incorta releases and are using ADLS storage must create a copy of the IncortaNode/hadoop/etc/hadoop/core-site.xml file under IncortaNode/spark/conf/.

Important notes

Spark 3.3 supports Python versions from 3.7 to 3.10.

Important

If you have multiple Python binaries on the same machine, set the following attributes:

python.path to Python 3 path in the IncortaAnalytics/IncortaNode/node.properties
export PYSPARK_PYTHON = /usr/bin/python3 in IncortaNode/notebooks/services/<SERVICE-ID>/conf/zeppelin-env.sh

PySpark requires Pandas 1.0.5 or later.
PySpark requires PyArrow 0.12.1 or later.

Contact Incorta Support if you require further assistance.

Advanced session cookies security options

As an admin, you can now use the Cluster Management Console (CMC) to secure your session cookies by setting the SameSite attribute to restrict sending cookies to a first-party (same-site) context only and prevent the browser from sending cookies along with cross-site requests. You can also specify the Secure attribute to allow sending cookies to the server only when a request is made using an HTTPS scheme.

For more information, refer to Guides → Configure Server → Security.

Deactivate users

Incorta administrators (Super User / User Manager) can now deactivate Incorta analytics users instead of deleting them. Deactivating users will not delete users' data, instead, it will keep the data saved and usable for others. Also, deactivated users' scheduled jobs will remain active unless the administrator chooses to suspend them.

For more information, refer Tools → Security Manager.

Security roles enhancements

In this release, Incorta is enabling you to limit the roles of the User and Individual Analyzer user roles. Now you can disable downloading insights for these user roles using the Limit Insights Download option under Default Tenant Configurations > Security in the Cluster Management Console (CMC).

For more information about security roles, refer to References → Security Roles.