Release Notes 2022.4.0

Release Highlights

The 2022.4.0 release introduces multiple enhancements and major features. The main focus of this release is to empower users to get the most out of their data while maintaining a great user experience.

This release introduces the insight over result set labs feature to simplify creating complex queries in a two-step process without requiring much coding experience. The release also enhances the definition and support for global variables. It also introduces a new native PostgreSQL parser. It comes with multiple improvements to the data application Data Wizard, Component SDK, and public API. You will be able to easily identify column dependencies across different entities. The release allows also for better load time and performance during full load jobs. In addition, the release enhances security by upgrading the jQuery library that Incorta uses to version 3.6.0.

Warning

In case you use a data agent to allow Incorta to access your data, you must upgrade your data agent service to the new version available on the Cloud Admin Portal. Contact Incorta Support if you need help.

New Features

Insights over result sets

This Incorta Labs feature empowers Analyze users with a robust set of advanced analytic capabilities without the need to access the Schema Manager or Business Schema Manager or even have much coding experience.

Before this feature, building advanced analytic use cases, such as Top N, Top N%, level of detail (LOD) calculations, running totals, and moving average, required data engineers to pre-define the logic within complex materialized views (MVs) or in business schema views. However, with this feature, Analyze users can easily implement these use cases using the Analyzer. As an Analyze user, you first create a result set in the Analyzer (which is a runtime insight query in the form of a Listing table or Aggregated table insight) and then use this result set as a dataset to build the final insight.

This feature enables users to dynamically create complex queries in a two-step process with more control over the query’s order of operations. It enables users to apply conditional formatting with formulas and to create aggregate filters on Pivot tables. Result sets support dashboard prompts, presentation variables, applied filters, and built-in functions including level-based measures.

The following are some of the most significant use cases of this feature:

Comparative analysis
Percent contribution of rows and columns
Running totals
Rank analysis
Customer order frequency
Customer acquisition

For more information, refer to Visualizations → Insights over Result Sets.

Component SDK Improvements

Incorta is introducing major improvements to the Component SDK that aims to enhance the security and developers' experience as well as provide more capabilities to create different charts through a new and enhanced version of the SDK

Important

With the new enhanced version, the developer must re-download the Component SDK from GitHub and install it.

Note

Previously developed charts must be modified and recompiled to be compatible with the Incorta 2022.4.0 release and the new version of the Component SDK.

Incorta has also enhanced the development experience while writing code in the .tsx and definition.json files in the Component SDK projects by applying several improvements, such as improving the auto-complete feature and enhancing error messages in the Integrated Development Environment (IDE) and the Developer Visual in the Analyzer.

For more information, refer to the Incorta Component SDK.

Enhanced KPI Insight with Grouping Dimension tray

Incorta enhanced the KPI insight to include an optional Grouping Dimension tray. The KPI insight supports multiple tiles, one for each dimension value. It supports a prompt filter based on the dimension value driving the KPI tile in addition to drilling down capabilities to the next grouping dimension. For more information, refer to Visualizations → KPI

Data Applications Data Wizard enhancements

This release introduces multiple enhancements to the functionality and user interface of the data application Data Wizard. These enhancements ensure a better user experience for both trial users and enterprise customers. After connecting to the sample data or your data, the data application Data Wizard displays the schema diagram where you can review the data application data model and joins. This is limited to Quickstart data applications for now. However, for Enterprise data applications, you can still use the Schema Diagram Viewer to view the data model and joins.

For more information, refer to Data applications → Data Wizard

New public API send-report

Incorta introduces a new public API ‘send-report’ that you can use to send the dashboard as a report via email. For more information, refer to Tools → Public API

New Native PostgreSQL Parser

Incorta is introducing a new parser that is based on the native PostgreSQL library version 13. The new parser aims to support more PostgreSQL queries and functions, which will enhance the experience of the SQL Interface (SQLi) users, as well as the integration with external visualizations tools, such as Tableau and Microsoft Power BI. It also enhances query execution performance.

Important

The new parser accepts only valid PostgreSQL syntax.

Enhanced support for the PostgreSQL INTERVAL function

Incorta enhanced the support for the PostgreSQL arithmetic functions. The following is a list of all the newly added enhancements for the INTERVAL function:

Double in INTERVAL string, for example, INTERVAL ‘1.5 years’
PG_VERBOSE, SQL_STANDARD, ISO8601, and ISO_ALTERNATE
EXTRACT/DATE_PART with INTERVALs, EXTRACT
EXTRACT/DATE_PART with TIMESTAMP
TIMESTAMP/DATE + INTERVAL
TIMESTAMP/DATE - INTERVAL
INTERVAL + INTERVAL
INTERVAL - INTERVAL
INTERVAL * double_values
INTERVAL / double_values
INTERVAL * columns(numeric type)
columns(numeric type) / INTERVAL

Refer to the PostgreSQL documentation for more information and examples.

Additional Considerations

Support for TO_CHAR with INTERVAL types is not yet supported
CAST(Column as INTERVAL) works for string constants like CAST('1 year' as INTERVAL)
Incorta supports precision up to milliseconds only

Cloud storage cost optimization

This enhancement reduces the number of I/O operations on cloud storage, which leads to reducing the underlying cost. This is more effective with frequent incremental loads and schemas with a large number of tables.

Global Variable Enhancements

This release introduces multiple enhancements to global variables in regard to definition and support. You can define global variables using the query and queryDistinct functions in the Formula Builder instead of specifying a constant value. In addition, you can reference global variables in new contexts, such as formula columns and other global variables. Referencing a global variable in one of the new contexts (usually in the Formula Builder) is different from referencing it in the extraction queries or scripts of physical schema tables and MVs. In the Formula Builder contexts, precede the global variable name with a single dollar sign ($), rather than a double dollar sign ($$). The data type of a global variable value is no longer manually selected; it is determined by the query syntax and referenced columns.

For more information, refer to Concepts → Global Variable.

Support for tracking column dependencies across different entities

As a schema developer, you can preview the whole list of physical and business schema entities where a column is being referenced either directly or via another dependent column in only one step. This new feature aims to help schema developers better manage schema updates and load jobs.

When you select to display the column dependencies, you can find the following column-dependent entities on the list:

Columns and formula columns in physical schema objects: physical schema tables, MVs, Incorta SQL tables, Incorta Analyzer tables, and aliases
Columns and formula columns in business schema objects: business views, Incorta Analyzer views, and Incorta SQL views
Join columns
Runtime security filters
Load filters

This list is not limited to the entities where the column is directly referenced. All entities that can be impacted by the update or deletion of this column are listed.

For more information, refer to Tools → Table Editor → Preview column dependencies.

Changing the default behavior of the PK-Index Calculations on Full Load

Starting this release, the Primary Key (PK) index calculations during a full load job will be disabled by default for newly created physical tables and MVs with one or more key columns. The new behavior will not affect existing tables and MVs. Enable the Enforce Primary Key Constraint toggle in the Table Editor if the source of your table or MV has duplicate data that you do not want to bring to Incorta. Disabling this option enhances the load time and performance and memory requirements during full load jobs. However, this may cause data redundancy.

jQuery Upgrade

Incorta upgraded the jQuery library used from version 2.1.1 to version 3.6.0 to enhance its security.

Additional Enhancements and Fixes

In addition to the new features and major enhancements mentioned above, this release introduces some additional enhancements and fixes that help to make Incorta more stable, engaging, and reliable.

Physical Schema

Fixed an issue with rejected rows due to data type mismatch in physical schema tables that use CSV files as a data source, in which the number of rejected rows was different from the actual ones in the downloaded CSV file
Fixed an issue that caused a warning requiring a full load of schema not to appear when editing a function or encrypted column
Resolved an issue where schema tables with a trailing space failed to load
Resolved an issue for certain tables that caused requesting incorrect load type, especially, a full load rather than a staging load
Fixed an issue in which selecting a deleted object in the load status menu would redirect the user to an incorrect page and would not display an error message denoting that the object is missing
Fixed an issue in which the Save without discovery option was erroneously available for Incorta SQL tables
Fixed an issue where a user could erroneously select to add Incorta SQL table data sources to multi-source tables
Resolved an issue in which an encrypted measure or dimension column would not have its encryption status set to false when the column’s function was changed to a key
Resolved an issue that caused loading Incorta SQL and Analyzer tables to fail with “Snapshot tables do not support full load” error
Fixed an issue where the geo column type would not display in the column type menu of a table
Resolved an issue where a table using drill-downs would display with all categories expanded
Fixed an issue where schema loads involving the Data Agent could take longer than required to verify query execution

Scheduler

Fixed an issue in which scheduled dashboard folders exported during the export tenant process were not imported with the tenant
Fixed an issue with the Send Now option in which dashboards were not sent if the Pause Scheduled Jobs toggle was enabled in the Cluster Management Console (CMC)

Business Schema

Fixed an inconsistency issue in the formula column data type between Business Schema View and Incorta Analyzer view
Fixed an issue where the hint was not displayed if a user attempted to save a business view that referenced columns from multiple business views
Resolved an issue where saving a listing table insight as a business schema view resulted in wrong display of the listed column names

Visualizations

Enhanced the Aggregated Table visualization such that a column sort performed in the Analyzer is also reflected in the insight when downloaded to Excel or CSV
Resolved an issue in the Aggregated Table visualization in which multi-row entries in the grouping dimension were trimmed
Fixed an issue in which downloading and sending insights as PDF and HTML resulted in an error
Fixed an issue in the Aggregated Table visualization in which conditional formatting did not appear for subtotals when the Merge Columns and Collapsed insight settings were enabled
Resolved an issue where text in a right-to-left language overlapped on Organizational charts

Dashboards

Fixed an issue in which the dashboard was not formatted correctly when emailed to an Outlook client
Resolved an issue where updating a bookmark could result in an error
Fixed an issue where sending data to a Google sheet could fail if a filter was applied to the dashboard
Fixed an issue in which filter values may not appear in the UI when using the Between operator
Fixed an issue that prevented pagination in SQL-view-based insights
Resolved an issue where an error could occur in rendering an insight when applying a filter

Analyzer

Enhanced the performance of joining rows between base tables and target tables
Fixed an issue where enabling Merge Columns in the Analyzer caused an isight rendering error
Resolved an issue where column data would not display when the last column in the data source query had an alias name

Materialized Views and SQLi

Enhanced the logs by adding the column name to thread names
Resolved an issue in which running notebook resulted in an error when cluster mode was enabled
Resolved an issue where in some cases SparkSQL queries on a Notebook could fail
Fixed an issue where auto-complete for spark Python in a Notebook would not suggest the correct function
Resolved an issue where fetch size settings were not respected when using an ODBC driver in certain configurations
Resolved an issue in which saving Notebooks with two paragraphs in MVs resulted in loss of paragraphs

Cluster Management Console (CMC)

Resolved an issue where a backed-up tenant would not properly be restored
Fixed an issue where restoring and overwriting a tenant could result in only deleting the tenant
Resolved an issue where the path of a restored tenant would not display in the tenant details
Resolved an issue in which the CMC logs were not being saved
Fixed an issue that caused importing old tenants to fail

Miscellaneous

Resolved multiple Japanese translation issues in the Content Manager and the Scheduler
Resolved an Incorta Metadata error when loading the SQL_Audit table
Fixed an issue that caused Incorta to fail to access a tenant created on an Azure Data Lake Storage (ADLS) that existed on the Azure China cloud
Fixed an issue with external session variables that referenced an SQL Server data source, which caused testing the query to fail
Resolved an issue where guide videos in the load data window were not available
Fixed an issue in which an Oracle metadata DB tenant could not be imported to a MySQL metadata DB
Fixed an issue where some median formulas could cause an unnecessary increase in the log file size
Enhanced the security by upgrading the Apache Log4j2 used in different components to the latest version 2.17

Upgrade Considerations

After upgrading to the 2022.4.0 release, all existing global variables referenced in physical schema tables or MVs will function as expected. However, if you try to test these variables, it will throw errors. In addition, you must use a query or queryDistinct expression when updating any of the existing global variables.

Known Issues and Workarounds

The following is a list of the known issues in this release, with a workaround if it is available:

No validation on functions is triggered in the Formula Builder when creating global variables.
Global Variables will not appear on the list of variables when referenced in a column individual filter, filter option, or applied filter, that is when you type$ in the search values box. However, you can add the global variable manually, if applicable. In Query, Contains, and Starts With are examples of functions that accept a global variable as a filter value to be added manually.
Global Variables that return a list of values are not functioning properly when referenced in MVs or individual filters
Text in right-to-left languages is aligned left on Organizational charts. However, it is read normally from right to left.
Changing the script language used for a loaded MV from Spark SQL to Incorta PostgreSQL throws errors
Using the DISTINCT operator on a measure in a Pivot table insight that is built over an Incorta SQL View throws errors
Exporting a tenant, from a 2022.4.0 cluster, with a global variable that uses the newly supported syntax (query or queryDistinct functions), and then importing it to the same or another 2022.4.0 cluster will cause the Edit Global Variable dialog to show an empty Value. You can manually edit the value to enter the query expression again.
Incremental load jobs of tables with SQL-based data sources, such as Microsoft SQL Server and Oracle, will fail if the Incremental Script has multiple conditions with the ? option to retrieve new and updated data.
Formula columns that reference global variables are evaluated as string in the following cases:
- Importing a tenant
- Restoring a tenant
- Starting up a server or a cluster
Global variables with date or timestamp data types are not evaluated properly when referenced in the Analyzer.
After changing the data source of an object in an imported physical schema or tenant to an Incorta SQL Table or changing an Incorta SQL Table in an imported physical schema or tenant to another object type, MV for example, the Disable Full Load option will be automatically enabled, which will lead to skipping these objects in the next schema full load jobs.
- You need to either disable the feature manually for objects that support disabling full loads (physical schema tables and MVs with incremental load enabled) or recreate the objects.

Maintenance Packs

Maintenance Pack 2022.4.1

Fixed an issue that caused changing the order of columns in a loaded physical schema object not to be correctly reflected in the Analyzer
Fixed an issue where applying conditional formatting to a measure column in a tabular insight with dynamic hierarchy caused some values in the measure column not to appear in the Analyzer
Fixed an issue where using the Search box in the dashboard resulted in an error
Fixed an issue that caused the Fixed Placement setting option not to properly function
Fixed a case while importing a tenant where the Next button was available although the Path field was empty
Fixed a UI issue that caused variables and functions to appear in blue in the Functions and Variables panel of the Formula Builder