Release Notes 5.2

Release Highlights

The goal of the Incorta 5.2 release is to introduce new features, new connectors, and major enhancements, such as the Insights over Analyzer result sets, Incorta SQL Views, Oracle Enterprise Performance Management (EPM) connector, FTP and SFTP as data destinations, Cluster Management Console (CMC) Monitoring tool, and much more.

Incorta is now supporting the backslash as an escape character in the formulas as well as the TO_NUMBER PostgreSQL format function. In addition, this release introduces a new and improved Schema Manager and Designer.

Incorta has also enhanced several visualizations, such as the bubble and advanced map visualizations. Incorta is now also supporting Apache Tomcat 9.0.54.

These new features and enhancements improves the platform’s analytical capabilities, data management, and performance. It also enables you to further process your data and maximize your benefit from it.

Upgrade Considerations

Before upgrading to this release you must consider the following points:

In the 5.2 release, Incorta is only supporting the MySQL Metadata database, Linux OS, and NFS as a file system. For more information, contact your Incorta customer success manager. For more information, refer to Metadata Database Migration.
Due to the changes in the directory structure, on-premises customers who use external Spark must manually add the delta-core JAR file compatible with their Spark version and the incorta.delta-1.0.jar file to the incorta directory under the external Spark installation path. For more details, see Delta Lake compatible catalog folder structure.
When upgrading to the 5.2 release, existing formula columns with a backslash \ as part of the formula text will display no data. You must precede each backslash with another backslash. For more information, refer to Backslash support as an escape character.

Warning: Incorta Intelligent Ingest

For now, the 5.2 release does not support Incorta Intelligent Ingest. However, a subsequent maintenance pack of the 5.2 release will support it soon.

New Features

Insights over result sets

This feature empowers Analyze users to create insights based on intermediate result sets without leaving Analyzer.

Before this feature, building advanced analytic use cases, such as Top N, Top N%, level of detail (LOD) calculations, running totals, and moving average, required data engineers to pre-define the logic within complex materialized views or in business schema views.

As an Analyze user, you first create a result set in the Analyzer (which is a runtime insight query in the form of a Listing table or Aggregated table insight) and then use this result set as a dataset to build the final insight.

This feature enables users to dynamically create complex queries in a two-step process with control over the query’s order of operations. It enables users to apply conditional formatting with formulas and to create aggregate filters on Pivot tables. Result sets support dashboard prompts, presentation variables, applied filters, and built-in functions including level-based measures.

Note: The Insights over result sets is an Incorta Labs feature

An Incorta Labs feature is experimental and functionality may produce unexpected results. For this reason, an Incorta Lab feature is not ready for use in a production environment. Incorta Support will investigate issues with an Incorta Labs feature. In a future release, an Incorta Lab feature may be either promoted to a product feature ready for use in a production environment or be deprecated without notice.

The following are some of the most significant use cases of this feature:

Comparative analysis
Percent contribution of rows and columns
Running totals
Rank analysis
Customer order frequency
Customer acquisition

For more information, refer to Visualizations → Insights over Result Sets.

Download to Excel support for user interface (UI) settings

Download of a dashboard or insight to Excel will support UI level settings, which include totals and subtotals, and column formatting.

Support for date and timestamp data types in CASE and DECODE conditional statements

The case and decode conditional statements in the Analyzer now support the date and timestamp data types in their signature. For more information, refer to Built-in Functions → case and Built-in Functions → decode.

Visualization enhancements for Donut Total and Table style and appearance

The Analyzer now supports the following features in the Settings panel:

A new Total setting for the Donut visualization, which enables you to show the total of a measure in the white space inside of the Donut. In addition, you can also choose whether to hide or show the measure name using the setting Show Measure. For more information, refer to the Visualizations → Donut.
A new Style and Appearance section for configuring tables, such as font size, color, and alignment of headers and values. For more information, refer to the Settings Panel section in the Listing Table, Aggregated Table, and Pivot Table documents.

Bubble Chart visualization enhancements

Incorta has enhanced the Bubble chart visualization to allow both aggregated and listed result sets. You can now use dimensions and measures in the Horizontal and Vertical trays, add a measure to the Size by tray to precisely size the bubble, and group the plotted points by the coloring dimension.

For more information, refer to Visualizations → Bubble.

Advanced Map visualization enhancements

In this release, the Analyzer supports the following enhancements to the Advanced Map visualizations:

You can drag and drop columns with Geo data types only to the Geo Attribute tray.
A new Geo icon for the Geometry column in the Analyzer’s Data panel for Advanced Map visualizations. This new Geo icon distinguishes geometry columns that use custom shapes from string columns.
In the Color By tray, the default Aggregation is now set to MIN instead of COUNT for columns of type String.
A new Custom Label tray is now available for GEO DATA that applies to Geo Attribute and Geospatial (Lat/Long) entities, in which the tray is only visible when the Data Label option is enabled in the Layer settings. The value of the column in the Custom Label tray overrides the default label shown for this layer on the Advanced Map insight.

For more information, refer to Visualizations → Advanced Map.

Backslash support as an escape character

The Engine now supports using the backslash mark \ as an escape character that you can use within a string in a formula to treat the next character as literal text. For strings that you enclose in double quotes and include double quotation marks within, you must precede each quotation mark that you want to include in the string with a backslash \. The same applies to single quotation marks within a string enclosed in single quotes. In addition, to include a backslash in a string, precede it with another backslash as an escape character.

Here is an example:

The formula:

concat(
   "Place a string between two double quotes: \"TEXT\"",
   ' or two single quotes: \'TEXT\'.', "Use a backslash \\ as an escape character."
)

The result:

Place a string between two double quotes: "TEXT" or two single quotes: 'TEXT'. Use a backslash \ as an escape character.

Important

After upgrading to the 5.2 release, existing formula columns with a backslash \ as part of the formula text will display no data. In addition, when you try to save a formula with a single backslash in the Formula Builder, this will show errors. You must precede each backslash by another backslash. The same errors appear with one single quote in a string enclosed in single quotes and one double quote in a string enclosed in double quotes without a preceding backslash.

New format options for the Date column

You can now format the column of a Date Part in the Grouping Dimension tray. When you set the Date Part to Quarter or Month in the Properties panel, a Format dropdown list appears to select your preference. Here are the formatting options:

For Quarter Format: No Format (1) or Prefix (Q1)
For Month Format: No Format (1), Prefix (M1), Short (Jan), or Long (January)

Dynamic Group By for multiple columns in Aggregated Tables

The Aggregated Table visualization now allows Dynamic Group-by for multiple dimension columns. To learn more, refer to Visualizations → Aggregated Table.

Incorta SQL Views in Business Schemas

Incorta now supports SQL as a type of Incorta View in addition to the Incorta Analyzer View in Business Schemas. This new feature enables you to create complex SQL queries in a new view. Using the provided SQL editor, you can write simple and complex SQL queries, format these queries, and run them to see the output before you save the view.

Note: The Incorta SQL Views is an Incorta Labs feature

For more information about the Incorta SQL View, refer to Concepts → Incorta SQL View.

Infrastructure Monitoring tool in the Cluster Management Console

Incorta is introducing a new Infrastructure Monitoring tool in the Cluster Management Console (CMC) that enables you to view your infrastructure utilization graphs and insights per cluster. For example, you can monitor the Metadata Database, Spark, Zookeeper, and Notebook services. For more information, refer to Tools → CMC Monitoring.

Google Cloud Storage Connector

A new connector to support the use of Google Cloud Storage (GCS) as a data lake data source is now available. To learn more, refer to Connectors → Google Cloud Storage.

Oracle EPM connector

Now you can get financial data from Oracle Enterprise Performance Management (EPM) inside Incorta using the new Oracle EPM connector. This connector enables data extraction for consolidation, planning, and budgeting applications.

For more information, refer to Connectors → Oracle EPM.

Important

The Oracle EPM connector is a preview connector. Contact Incorta Support if you want to use it in production environments.

SAP ERP Connector enhancements

The SAP ERP connector now includes the following:

A Datasource Timezone property, which preserves the date, time, or timestamp from the data source to Incorta.
Support for different full and incremental load queries with the same number of columns. The Full Load Derived Columns and Incremental Load Derived Columns properties are available to add columns post-extract for cases in which the full load query and incremental load query have differing numbers of columns.
The option to choose the backend database fetch method for full and incremental loads, which now includes OPEN_CURSOR for improved performance.
New Chunking Column Type options:
- Chunking by Year and Period columns
- Full Load by Filter
Improved data extraction performance in which business rules are processed in Incorta by default, with the following features:
Currency decimal shift for amount fields associated with currencies that do not have decimals
Timestamp data type identification and conversion to a date and time format

For more information, refer to Connectors → SAP ERP.

Oracle Cloud Applications connector Maximum Cache Size

The Oracle Cloud Applications connector now includes a Maximum Cache Size property. This property can be used to limit the size of downloaded files to prevent running out of disk space.

For more information, refer to Connectors → Oracle Cloud Applications.

Data Lake connector file exclusion

The data lake connectors now have an Exclude table data source property. This property allows you to exclude all files with names that match the specified pattern from the load process.

For more information, refer to Connectors → All → Data Lakes.

Automatically force load jobs stuck in the interruption process to abort

In this release, Incorta automatically aborts any load job that is interrupted in the extraction phase and exceeds the configured interruption time interval. This removes the stuck physical schema from the Schema Pool, which allows users to start a new load job of the related physical schema and allows loading other physical schemas as well. The status of this load job is Aborted.

The default time interval to automatically abort stuck interrupted load jobs is 60 minutes. You can ask the Incorta Support team to edit the service.properties file that exists in the directory of each Loader Service and define another time interval in minutes. The following key controls the time interval: abort.interrupted.job.wait.time.

Different incremental load strategies for Materialized Views

This release supports two different incremental load strategies for MVs. In incremental load jobs, the Loader Service instructs Spark to fetch only new and updated records. You can load MVs incrementally depending on one of the following:

The MV’s last successful transformation time; available in previous releases
The maximum value of a column in the MV

For more information, refer to Concepts → Materialized View → Incremental load strategies.

Limited access to the Business Schema Manager for Analyze and Individual Analyzer users

Starting this release, users with only the Analyze User or Individual Analyzer roles will have limited access to the Business Schema Manager where they can view a list of business schemas without the need to be assigned the Schema Manager role. They can also export shared business schemas and view their description and sharing configurations.

These users can also access the runtime business views in the Business Schema Designer View Mode to view the runtime business view columns and invoke the Analyzer to explore their data and create insights.

Global variable support in MVs

In this release, you can reference global variables in materialized views when you use the Notebook Editor or the Query Builder to add or edit the MV script (or incremental script) using any supported language. This enables you to centralize the constant values that you reference in multiple MVs, which saves time and effort and ensures data consistency across all related MVs.

To reference a global variable, precede the variable name with double dollar signs, for example, $$gvar_name. When referencing string, date, and timestamp global variables, you must use single or double quotes depending upon the MV script language. For Spark Scala, use double quotes; for other languages, use single quotes, for example '$$string_gvar_name'.

Important

Global variables referenced in MVs are evaluated when validating the MV data source and when loading them. Thus, when you edit the value of a global variable, you must perform a full load of the related MVs.

For more information, refer to Concepts → Global Variable.

Notebook enforcement of physical schema permissions

Notebooks will now enforce the physical schema access control permissions. In a notebook, you will only be able to read a physical schema table and run queries against it if it is shared with you.

New security password for exporting and importing tenants in the CMC

Incorta is adding an extra security feature to protect your sensitive data, such as the credentials of your data sources, when you export a tenant using the CMC.

A new step is added when you export a tenant, where you can set a security password to export sensitive data. If you choose to export a tenant without a security password, you will have to re-enter all of your data source credentials after importing the tenant.

For more information, refer to CMC Tenant Manager.

Data type lock in a multi-source table

Starting with this release, a schema designer can lock the data type of one or more output columns in a multi-source table so that the data type will not be automatically cast or updated when validating data sources and inferring the output columns in the following cases:

Accessing the respective multi-source table in the Table Editor
Adding a new data source to a multi-source table
Editing one of the data sources in the Data Source dialog and validating all data sources
Deleting one data source from a multi-source table with more than two data sources

For more information, refer to Tools → Table Editor → Manage data sources.

Delta Lake compatible catalog folder structure

This release introduces a new catalog folder structure that is compatible with Delta Lake. This new enhancement results in changes to the Shared Storage directory structure, the output of a compaction (deduplication) job, and the way compacted-parquet consumers access the compacted segments.

The new compaction (deduplication) mechanism reduces the I/O operations during a load job and saves disk space.
Only Parquet files that have duplicates are rewritten to create compacted segments.
Compacted segments are saved per compaction job under a new directory in the source area under the object directory: the _rewritten directory.
Extracted Parquet files with no duplicates are no longer copied to the compacted segments directory (_rewritten).
Different versions of the compacted Parquet files can exist under the _rewritten directory.
A group of metadata files is generated per compaction job in Delta Lake file format to point to all Parquet files (whether extracted or rewritten) that constitute a compacted version. These metadata files are saved to a new directory, _delta_log, that exists also in the source area under the object directory.
Consumers of compacted Parquet files (MVs, SQLi on Spark port, internal and external Notebook services, and the Preview data function) will use the latest <CompactedVersionID>.checkpoint.parquet metadata file to find out which Parquet file versions, whether extracted or compacted, to read data from. In addition, the Cleanup job checks the same file before deleting the unused compacted Parquet versions.

Delta Lake Upgrade Considerations

To read the Delta Lake metadata files, Spark can use either its native Delta Lake Reader (Default) or the Incorta Custom Delta Lake Reader.
- To use the Incorta Custom Delta Lake Reader, Spark must include the incorta.delta-1.0.jar file.
- To use its Delta Lake Reader, Spark must include the appropriate delta-core JAR file.
  - Spark v3 is compatible with the delta-core_2.12-0.7.0.jar file.
  - Spark v2 is compatible with the delta-core_2.12-0.6.0.jar file.
- Environments that use Apache Spark bundled with Incorta will have the appropriate delta-core JAR file and the incorta.delta-1.0.jar file added to their cluster (/home/incorta/IncortaAnalytics/IncortaNode/spark/incorta) as part of the upgrade process.
- On-premises customers who use external Spark must manually add the delta-core JAR file compatible with their Spark version and the incorta.delta-1.0.jar file to the incorta directory under the external Spark installation path. Contact Incorta Support for help.
- Turning on the Enable a custom Delta Lake Reader option in the CMC enables using the Incorta Delta Lake Reader.
After upgrading to the 5.2 release, consumers will continue to read from the old compacted directory of an object until the first full or incremental load job (loading from source) on this object or the physical schema that requires compaction. The new folder structure will be created gradually, that is, it will be created separately for each object after the first full or incremental load job that requires compaction.

Important

After the 5.2 release upgrade, the first full or incremental load job of each object may take a longer time to create the new structure and perform full compaction of all the required Parquet files.

For more information, refer to Data Ingestion and Loading → Enhancements to the compaction process.

FTP and SFTP servers as Data Destinations

Incorta now supports FTP and SFTP servers as a data destination, where you can export dashboard data. You can connect to FTP servers using the FTP or FTPS protocols.

The following are the supported visualizations that you can send to an FTP or SFTP server data destination:

Note

You can export Pivot table insights to the XLSX file format only.

For more information, refer to Concepts → Data Destination.

Support for the TO_NUMBER PostgreSQL format function

This release supports the TO_NUMBER(string, format) PostgreSQL format function to convert various string data types to numbers.

The function is available for a PostgreSQL script in a materialized view and for a SQL query that uses the SQL interface (SQLi) and Apache Spark.

The following table illustrates the supported formats:

Format	Description
`9`	Numeric value with the specified number of digits
`0`	Numeric value with leading zeros
`.` (period)	Decimal point
`D`	Decimal point that uses locale
`,` (comma)	Group (thousand) separator
`FM`	Fill mode, which suppresses padding blanks and leading zeroes
`PR`	Negative value in angle brackets
`S`	Sign anchored to a number that uses locale
`L`	Currency symbol that uses locale
`G`	Group separator that uses locale
`MI`	Minus sign in the specified position for numbers that are less than 0
`PL`	Plus sign in the specified position for numbers that are greater than 0
`SG`	Plus `/` minus sign in the specified position
`RH`	Roman numeral that ranges from 1 to 3999
`TH` or `th`	Upper case or lower case ordinal number suffix

Note

The previous string formats also apply to the TO_CHAR() PostgreSQL function.

Schema Designer and Table Editor New User Interface (UI)

This release introduces a new UI for the Schema Designer and the Table Editor. The new UI enriches the user experience and facilitates managing objects and joins in a physical schema. The new UI has the same look and feel as other components, such as the Analyzer and the Catalog (Content Manager), which provide a consistent user experience. For example:

A tab-based view for tables and joins, which reduces the need to scroll vertically across multiple tables to get to the joins section.
Different filter, search, and input options in the Schema Designer and Tables Editor, which enable you to do the following:
- Search for tables and joins
- Filter the table list by the object type and the performance optimization status
- Filter the object columns by the column type, function, and encryption status
- Show or hide the schema name in the join list
- Use the available visual indication to find joins with join filters
- Use the date picker when defining a join filter

For more information, refer to Tools → Schema Designer and Tools → Table Editor.

Export and Import an Incorta Notebook

This release supports exporting and importing Incorta Notebooks in Materialized Views (MVs). You can now upload a new Notebook to replace the existing one, download the current Notebook, and import a new Notebook inside a specific physical schema. After performing an import action, you must select Done to save your changes.

Note

You can only import JSON files to the Notebook. For more information, refer to Tools → Notebook Editor.

Enhanced data loading into memory

This release introduces a new feature to enable parallel reading of parquet segments to enhance the performance of loading data into the Analytics Service memory, especially when rendering dashboard insights for the first time.

This feature is beneficial in the following situations:

The ratio of unique values in a column is low.
Insights require reading multiple columns at the same time.
There is a large number of parquet segments or rows.
Insights mainly reference integer or long data columns.

This feature is currently disabled by default. To enable it and manage its related settings, contact Incorta Support to edit the engine.properties file in both the Analytics Service and the Loader Service directories.

NetSuite SuiteQL Connector to support REST

A new NetSuite SuiteQL connector is now available that supports REST. To learn more, refer to Connectors → NetSuite SuiteQL.

New Connectors tab for managing CData connectors in the CMC

A tenant administrator can now control the usage of CData connectors through the new Connectors tab in the CMC during an install or upgrade process.

Here are the steps to manage connectors:

Sign in to the CMC.
In the Navigation bar, select Clusters.
In the clusters list, select a Cluster name, and then select the Connectors tab.
In the Actions bar, select Manage Connectors.
In the Configure Cdata Connectors dialog, select Allow to enable the use of a connector within a tenant, or select Remove from list to disable its use.
Select OK.

Enabling a connector does not require restarting the Analytics and Loader nodes. However, removing a connector requires restarting both nodes.

Data Agent support for version option in the agent script

You can now check your Data Agent version in the agent script by running ./agent.sh version on Linux or agent.bat version on Windows. For more information about how to enable and download the Data Agent, refer to the [Tools → Data Agent](/5.2/tools-data-agent/#enable-the-data-agent-for-an on-premises-incorta-cluster) document.

Pre-SQL support for Oracle and MySQL connectors

This release introduces a new feature that allows running SQL statements or calling stored procedures before executing the original extraction query and incremental query for a MySQL or Oracle data source during a load job.

In this release, the Data Source dialog has a new toggle, Pre-SQL, to enable this feature for a given data source. After enabling this toggle, use the Pre-SQL statement box to invoke the Query builder to enter the Pre-SQL statement or call the stored procedure you want.

For example, CALL app1_ctx_package.set_empno(11); where set_empno is a procedure that sets the employee number to 11.

The data source database management system determines the accepted statements and supported syntax.

This feature is useful when you need to:

Set the security context of the query before extracting data
Run stored procedures
Update the data source before extraction
Create a temporary table before executing the extraction query
Delete old records in the source so that the source includes only the latest records

Note

If the Pre-SQL statement fails during a load job, the object data extraction and load fail and an error message appears. Logs also will contain the failure reason.

Notebook support for using specific paragraphs in the MV script

You can now choose specific paragraph(s) to create an MV script by selecting Include in MV script (+ icon) in the Notebook Editor. This feature is available for all supported languages: SQL, PostgreSQL, Python, R, and Scala. However, if multiple languages are included in the same Notebook, only the paragraphs written in the default Notebook language can be selected.

If Include in MV script is not selected, then by default:

For SQL and PostgreSQL: only the last paragraph will be included.
For Python, R, and Scala: none of the paragraphs will be included and the script will be empty.

PostgreSQL MV support for an equivalent SparkSQL query with copy and refresh options

You can now generate a read-only SparkSQL query that is equivalent to your PostgreSQL query, and then copy it by selecting Copy to Clipboard in the Edit Query dialog of a PostgreSQL Materialized View table. Also, after editing a PostgreSQL query, you can select Refresh to show the updated SparkSQL query.

Encryption and decryption support in Scala Materialized Views

In this release, you can reference an encrypted column in a materialized view that is defined using Spark Scala. In addition, you can turn on the encryption property of a column in a Scala MV using the Table Editor.

When reading an encrypted column referenced in a Scala MV or an encrypted column in a Scala MV, while rendering a dashboard, for example, decrypted data is correctly displayed.

Parallel enrichment of independent MVs within the same physical schema

This release improves the prolonged transformation phase of independent MVs within the same physical schema. During a schema load job, the Loader Service will start the enrichment of multiple independent MVs in parallel whenever resources are available. This tends to reduce the MV in-queue time and the enrichment time as a whole. For MVs with cyclic dependencies, ordering these MVs still depends on the alphabetical order of the MV names.

Enhanced Dashboard query performance

In this release, Incorta has enhanced the performance of dashboards by adding a temporary cache layer, which will cache the populated result-set during the loading of an insight. Any concurrent requests for the same temporary table will use the cached temporary table and only one insight can populate it.

As an Incorta user, you can now login to Incorta Analytics using Single Sign-On (SSO). A new Use Single Sign-on button is added to the Incorta Analytics login page to facilitate the login process.

The CMC administrator must enable and configure your SSO provider per tenant. For more information on how to configure SSO and start using it, refer to the Secure Login Access and Start guides.

Interrupt long-running dashboards that block sync processes

This release introduces a solution for long-running dashboard queries that block synchronization processes, and accordingly other queries, without a major effect on the dashboard performance. The solution includes interrupting long-running queries that block a synchronization operation after a configured time period starting from acquiring the read lock by the running query.

This feature is disabled by default. To enable this feature or change the configured time, edit the the engine.properties file that exists in the Analytics Service directory to add the following two keys:

Key	Description	Data Type	Value
engine.interrupt_query_by_sync	Enable or disable the interrupting long-running queries feature	Boolean	● true ● false (default)
engine.query_render_time_before_interruption_by_sync	Set the time (in minutes) to wait since the running query that blocks a sync process acquires the read lock before interrupting the query	Integer	Number of minutes The minimum value is 0. The default value is 10.

Note

Although multiple operations acquire read lock on resources (physical schema objects and joins), such as searching in a dashboard, query plan discovery, and synchronization in the background, the solution handles only read locks acquired by dashboard queries that the engine runs. This includes the following:

Rendering a dashboard
Exporting a dashboard
Downloading a dashboard tab or an insight
Sending a dashboard, dashboard tab, or insight to a data destination
Sending a dashboard via email
Running a scheduled job to send a dashboard via email or to a data destination
Rendering a SQLi query that uses the engine port: the default is 5436

Depending on the interrupted process, whenever a query is interrupted, a message is displayed in the Analyzer, sent to the user via email, or displayed in the SQLi audit files. The message denotes that the query is interrupted because the underlying data is being updated.

Solution limitations

Due to the Java implementation to interrupt running processes and avoid major performance degradation, the interrupted query does not release the lock immediately. It may take some time until it hits an interruption check first.
This solution does not apply to synchronization that runs in the background.

Define the number of data frame partitions resulting from loading an MV

This release enables you to define the number of the data frame partitions resulting from loading a materialized view. In the MV Data Source dialog, you can add the spark.dataframe.partitions property and set the number of the data frame partitions as appropriate. If required, the Loader Service will perform either a coalesce or reparation operation to create the required number of data frame partitions.

Loader memory reduction for encrypted columns

This release introduces a new mechanism when loading and reading encrypted columns. The Loader Service no longer creates snapshot files for encrypted columns, only parquet files. In addition, the Loader service will load only required columns into its memory instead of loading all the table columns. The Loader and Analytics services will now read encrypted columns from parquet files. This new mechanism ensures both reducing the memory used by the Loader service during load jobs and saving disk space. This mechanism also enhances the load time when loading data incrementally because the column data will not be evicted from memory and reloaded again; only new data is loaded.

Note

The new mechanism results in a minor degradation in the Analytics service performance when reading columns (while discovering them or rendering dashboards).

Important

Changing the encryption status of one or more columns in a physical schema table or MV requires performing a full load for the related object.

A new version of the Excel Add-in

Version 16.8 of the Excel Add-in is now available. Here are some key features of the new version:

A new user interface that is consistent with the Incorta Analyzer
Support for Aggregated Tables in addition to Listing and Pivot Tables

For more information, refer to External Integrations → Excel Add-in.

Additional Enhancements and Fixes

In addition to the new features and major enhancements mentioned above, this release introduces some additional enhancements and fixes that help to make Incorta more stable, engaging, and reliable.

Home Page

Enhanced the cards shown under the Learn More section on the Home page to be user-friendly. Based on the user role, the most relevant cards are highlighted.
Fixed an issue in which long usernames on the Home page banner became truncated

Physical Schema and Business Schema

Enhanced the error message details that appear when loading a materialized view table for the first time while the full load option is disabled
Resolved an issue in which alias column names appear empty in tables
Fixed an issue with physical schema load from ADLS and GCS, in which incremental schema loads caused the load process to be stuck in the writing tables stage
Fixed an issue with the calculation of joins between non-derived tables when updating a physical schema
Fixed an issue in which schema load failed while trying to read parquet files
Fixed an issue in which viewing columns of a business view not created by the Analyzer using a database tool are not displayed
Resolved an issue where tables could intermittently not exit the extraction phase

Dashboards and Visualizations

Enhanced the performance of Listing tables with multiple grouping dimensions
Enhanced the dashboard performance after materializing formula columns
Enhanced the behavior of the default bookmark of the dashboard. Now when you add a default bookmark to a dashboard you can either choose to include the default prompts or exclude them.
Enhanced Advanced Map query performance by running separate queries for each map layer
Enhanced the query process of Excel for Aggregated, Listing, and Pivot tables to be able to see subtotals at a group level based on the insight level setting
Enhanced the chart legend and tooltips font size and color
Enhanced the Pie and Donut chart to increase the chart size, wrap long legend text, and decrease the white space in the visualizations
Enhanced the behavior of the default bookmark of the dashboard. Now when you add a default bookmark to a dashboard you can either choose to include the default prompts or exclude them.
Resolved an issue where subtotals in aggregated tables show as blank if the number of rows in the table equals or less than the page size
Resolved an issue in which subtotals in imported dashboards would be removed when the Subtotal setting is ON
Resolved an issue in which a join using base table is created incorrectly causing the dashboard to fail
Resolved an issue where text in a right-to-left language overlapped on Organizational charts
Enhanced the performance of dashboard rendering by increasing the number of threads in grouping and aggregation
Fixed an issue in which the dashboard was not formatted correctly when emailed to an Outlook client
Resolved an issue where an error could occur in rendering an insight when applying a filter
Fixed an issue in which downloading and sending insights as PDF and HTML resulted in an error
Fixed an issue in the Treemap visualization, in which drill down for measure columns does not work when you filter the dashboard using the coloring dimension
Fixed an issue in the Advanced Map visualization, in which drill down does not function properly
Fixed an issue in the Advanced Map visualization in which adding a Color By column to only one layer of a multi-layer insight caused unexpected behavior
Fixed an issue in dashboards with presentation variables, in which filtering these dashboards using session variables did not display any data
Fixed an issue in which sorting measure columns in Listing table insights did not sort correctly
Fixed an issue in which exporting tables to a PDF format did not draw the table gridlines
Fixed an issue with the drill down option in the Heatmap layers of an Advanced Map insight
Resolved multiple issues when exporting dashboards to PDF or HTML. Exporting dashboards to these formats now honors the status of the Append Timestamp check box and the exported file type.
- When clearing the check box, the export time will not be added to the name of the exported file or the .zip file.
- Dashboards exported to PDF are now sent to the export folder as PDF files rather than HTML files.
Fixed an issue in which filtering a dashboard that has a presentation variable with a valid session variable does not return data
Fixed an issue with sending dashboards via email in Excel or CSV format, in which the default value of a presentation variable is not equal to the session variable value in the email body
Fixed an issue in table visualizations in which gridlines were not maintained when the insight was downloaded to PDF
Fixed an issue in which copying a dashboard duplicated the tabs and insights with the same Globally Unique Identifier (GUID), which caused playing your favorite dashboards not to work properly when both the original and duplicated dashboards existed in the Favorites list
Fixed an issue in which filtering the dashboard that contained a presentation variable by a valid session variable did not return data
Fixed an issue in which the Send Now Dashboard option did not function properly when choosing PDF, HTML, or HTML File formats
Resolved an issue where columns could be truncated when exported to PDF
Fixed an issue in which Thousands Separator does not show for non-numeric columns in Listing Table visualization

Analyzer

Enhanced the performance of the ago and toDate time series analytic functions
Fixed an issue where queries fails to render and produce an error when using timestamp as a column dimension
Fixed an issue in which an error occurs while trying to open the Query Plan Viewer of a formula in the Aggregated Filter tray
Fixed an issue in which a join between a Materialized View table and an Alias table is broken in the insight’s Query Plan after updating the physical schemas
Fixed an issue that caused column headers to disappear in a table insight with zero rows when transposing the table
Fixed an issue with the Column Data Type drop down list in the Data panel in which logging in with the Arabic language did not translate the Column Data Type drop down list into Arabic

Data Agent

Fixed an issue with rendering tables in which the data agent has a delayed response from the data source, by setting a maximum threshold of 15 minutes for data agent retrials

Scheduler

Fixed an issue in which scheduled dashboard folders exported during the export tenant process were not imported with the tenant
Fixed an issue in which users where unable to delete emails from the CC and BCC properties of scheduled dashboards

Cluster Management Console (CMC)

Enhanced insight query performance by increasing Insight Max Groups UI Default from 500,000 to 1,000,000,000. You can edit this setting in the CMC for a specific tenant or in Default Tenant Configurations → Advanced.
Enhanced the name of the enable notifications option in Default Tenant Configurations to be Enable Sharing Notifications as this option enables you to share notifications with other users through emails
Resolved an issue in which deleting a tenant did not work and resulted in an error in the logs in Incorta version 4.9.5
Enhanced the security of migrating tenants with existing connectors. You are now required to enter the username and password for several connectors upon migrating tenants to your Incorta instance. For more information, refer to the Guides → Migrate
Enhanced the CMC logs to exclude password, client and app secret, API key, and passphrase values when changed

Integrations

Enhanced the query performance of the Tableau connector:
- Set an increased default fetch size of 10000
- Set the assumeMinServerVersion property to 10.0

Connectors and Data Destinations

Handled an error that might result from loading a table containing a numeric data type using the Presto connector
Resolved an issue in which chunking option was not available for the Presto connection in Incorta version 4.9
Resolved an issue in which special characters in dashboards name are removed when sent to a Google data destination
Fixed an issue with setting up Dropbox, Box, or Google Driver as a data source and data destination, when the security was implemented through a Proxy server and not through the Incorta Analytics hosting server.
- This fix requires using the proxy server information to define the Server URL Protocol, Server Name, and Server Port properties in the CMC → Tenant Configurations → Email. In addition, you must use the same information when registering the OAuth Redirect URI in the app to allow Incorta to access your cloud storage.
Fixed an issue where the Box connector would intermittently fail to extract
Resolved an issue in which loading data using the GoogleDrive connector, used to throw a Could not find file error

Materialized Views and SQLi

Enhanced the SQLi audit file by preventing writes of the same query multiple times and correcting the number of records in the audit file
Fixed an issue in which changing columns order in MVs only updates column label when exploring data via the Analyzer
Enhanced materialized view tables created using the Notebook Editor to support schema versioning
Fixed an issue in which SQLi queries from external business intelligence tools, such as Tableau and Power BI, resulted in a lock on schema loads
Fixed an intermittent issue with ODBC connections, in which the message contents do not agree with length in message type "N" error occurs
Fixed an issue that prevented MVs from reading data lake remote tables
Fixed an issue that caused the Audit files to be corrupted
Fixed an issue in which the decimal places were limited to 10 only in PostgreSQL
Fixed an issue in which schema, table, or column names that are longer than 63 characters did not appear in SQLi
Resolved an issue where query execution could fail to resolve business views due to an updated SQLi execution process

Security Manager

Fixed an issue in which an extra download link appears in the Insert Error details page, when you upload and execute an LDAP configuration file
Resolved an issue in which adding multiple users to a group worked only when users existed on the same page of the list

Miscellaneous

Fixed an issue in which embedding dashboards with prompt filters that use session variables in a Salesforce iFrame logs out of Incorta Analytics instead of displaying the dashboard
Enhanced the Loader Service logs to show both the physical schema and the object names if the Cleanup job fails to read the schema XML in the metadata database due to an invalid filter condition
Enhanced the security by upgrading the Apache Log4j2 used in different components to the latest version 2.17
Fixed an issue with sending dashboards via email in Excel or CSV format, in which the default value of a presentation variable is not equal to the session variable value in the email body
Fixed an issue in which UPDATE_ENTITY messages are flooding the loader service tenant log and consuming disk space
Resolved multiple Japanese translation issues in the Content Manager and the Scheduler
Fixed an issue where a query that runs on the DBVisualizer results in an error hen the business schema name is not valid on Incorta version 5.1.4
Fixed an issue in which the an interruption of the query parent threads did not also result in an interruption of the child threads
Fixed an issue in which sync dir for the command line using CLI caused a runtime exception
Fixed an issue in the login script by improving the detection of the underlying python version
Fixed an issue in which UPDATE_ENTITY messages are flooding the loader service tenant log and consuming disk space
Enhanced the logging of the PK index calculation process to decrease the size of its logs
Fixed an issue in which the Analytics Service did not release the read locks after rendering a dashboard
Fixed an issue in which logs consumed the disk space
Fixed an issue in which an SSO application could not sign in to an Incorta tenant when the tenant name is case insensitive in the URL
Fixed an issue with previewing data in Dark Mode
Fixed an issue with the Analytics Service in which the log displays error messages when a formula on prompt is created with a constant value
Fixed an issue with the search in the Individual Filter or Runtime Filter, in which a keyword preceded by a comma and space does not retrieve any search results
Fixed an issue in which the ago analytic function returned incorrect results when referencing business schema columns
Resolved an Incorta Metadata error when loading the SQL_Audit table