Release Notes 2022.4.0
The 2022.4.0 release introduces multiple enhancements and major features. The main focus of this release is to empower users to get the most out of their data while maintaining a great user experience.
This release introduces the insight over result set labs feature to simplify creating complex queries in a two-step process without requiring much coding experience. The release also enhances the definition and support for global variables. It also introduces a new native PostgreSQL parser. It comes with multiple improvements to the Blueprint Data Wizard, Component SDK, and public APIs. You will be able to easily identify column dependencies across different entities. The release allows also for better load time and performance during full load jobs. In addition, the release enhances security by upgrading the jQuery library that Incorta uses to version 3.6.0.
- Insights over result sets
- Component SDK improvements
- Enhanced KPI insight with Grouping Dimension tray
- Blueprints Data Wizard enhancements
- New public API send-report
- New Native PostgreSQL Parser
- Enhanced support for the PostgreSQL INTERVAL function
- Cloud storage cost optimization
- Global Variable Enhancements
- Support for tracking column dependencies across different entities
This Incorta Labs feature empowers Analyze users with a robust set of advanced analytic capabilities without the need to access the Schema Manager or Business Schema Manager or even have much coding experience.
Before this feature, building advanced analytic use cases, such as Top N, Top N%, level of detail (LOD) calculations, running totals, and moving average, required data engineers to pre-define the logic within complex materialized views (MVs) or in business schema views. However, with this feature, Analyze users can easily implement these use cases using the Analyzer. As an Analyze user, you first create a result set in the Analyzer (which is a runtime insight query in the form of a Listing table or Aggregated table insight) and then use this result set as a dataset to build the final insight.
This feature enables users to dynamically create complex queries in a two-step process with more control over the query’s order of operations. It enables users to apply conditional formatting with formulas and to create aggregate filters on Pivot tables. Result sets support dashboard prompts, presentation variables, applied filters, and built-in functions including level-based measures.
The following are some of the most significant use cases of this feature:
- Comparative analysis
- Percent contribution of rows and columns
- Running totals
- Rank analysis
- Customer order frequency
- Customer acquisition
For more information, refer to Visualizations → Insights over Result Sets.
Incorta is introducing major improvements to the Component SDK that aims to enhance the security and developers' experience as well as provide more capabilities to create different charts through a new and enhanced version of the SDK
With the new enhanced version, the developer must re-download the Component SDK from GitHub and install it.
Previously developed charts must be modified and recompiled to be compatible with the Incorta 2022.4.0 release and the new version of the Component SDK.
Incorta has also enhanced the development experience while writing code in the
definition.json files in the Component SDK projects by applying several improvements, such as improving the auto-complete feature and enhancing error messages in the Integrated Development Environment (IDE) and the Developer Visual in the Analyzer.
For more information, refer to the Incorta Component SDK.
Incorta enhanced the KPI insight to include an optional Grouping Dimension tray. The KPI insight supports multiple tiles, one for each dimension value. It supports a prompt filter based on the dimension value driving the KPI tile in addition to drilling down capabilities to the next grouping dimension. For more information, refer to Visualizations → KPI
This release introduces multiple enhancements to the functionality and user interface of the Blueprint Data Wizard. These enhancements ensure a better user experience for both trial users and enterprise customers. After connecting to the sample data or your data, the Blueprint Data Wizard displays the schema diagram where you can review the blueprint data model and joins. This is limited to Quickstart blueprints for now. However, for Enterprise blueprints, you can still use the Schema Diagram Viewer to view the data model and joins.
For more information, refer to Blueprints → Data Wizard
Incorta introduces a new public API ‘send-report’ that you can use to send the dashboard as a report via email. For more information, refer to Tools → Public API
Incorta is introducing a new parser that is based on the native PostgreSQL library version 13. The new parser aims to support more PostgreSQL queries and functions, which will enhance the experience of the SQL Interface (SQLi) users, as well as the integration with external visualizations tools, such as Tableau and Microsoft Power BI. It also enhances query execution performance.
The new parser accepts only valid PostgreSQL syntax.
Incorta enhanced the support for the PostgreSQL arithmetic functions. The following is a list of all the newly added enhancements for the INTERVAL function:
- Double in INTERVAL string, for example,
INTERVAL ‘1.5 years’
- PG_VERBOSE, SQL_STANDARD, ISO8601, and ISO_ALTERNATE
- EXTRACT/DATE_PART with INTERVALs, EXTRACT
- EXTRACT/DATE_PART with TIMESTAMP
- TIMESTAMP/DATE + INTERVAL
- TIMESTAMP/DATE - INTERVAL
- INTERVAL + INTERVAL
- INTERVAL - INTERVAL
- INTERVAL * double_values
- INTERVAL / double_values
- INTERVAL * columns(numeric type)
- columns(numeric type) / INTERVAL
Refer to the PostgreSQL documentation for more information and examples.
- Support for TO_CHAR with INTERVAL types is not yet supported
- CAST(Column as INTERVAL) works for string constants like CAST('1 year' as INTERVAL)
- Incorta supports precision up to milliseconds only
This enhancement reduces the number of I/O operations on cloud storage, which leads to reducing the underlying cost. This is more effective with frequent incremental loads and schemas with a large number of tables.
This release introduces multiple enhancements to global variables in regard to definition and support. You can define global variables using the
queryDistinct functions in the Formula Builder instead of specifying a constant value. In addition, you can reference global variables in new contexts, such as formula columns and other global variables. Referencing a global variable in one of the new contexts (usually in the Formula Builder) is different from referencing it in the extraction queries or scripts of physical schema tables and MVs. In the Formula Builder contexts, precede the global variable name with a single dollar sign (
$), rather than a double dollar sign (
$$). The data type of a global variable value is no longer manually selected; it is determined by the query syntax and referenced columns.
For more information, refer to Concepts → Global Variable.
As a schema developer, you can preview the whole list of physical and business schema entities where a column is being referenced either directly or via another dependent column in only one step. This new feature aims to help schema developers better manage schema updates and load jobs.
When you select to display the column dependencies, you can find the following column-dependent entities on the list:
- Columns and formula columns in physical schema objects: physical schema tables, MVs, Incorta SQL tables, Incorta Analyzer tables, and aliases
- Columns and formula columns in business schema objects: business views, Incorta Analyzer views, and Incorta SQL views
- Join columns
- Runtime security filters
- Load filters
This list is not limited to the entities where the column is directly referenced. All entities that can be impacted by the update or deletion of this column are listed.
For more information, refer to Tools → Table Editor → Preview column dependencies.
Starting this release, the Primary Key (PK) index calculations during a full load job will be disabled by default for newly created physical tables and MVs with one or more key columns. The new behavior will not affect existing tables and MVs. Enable the Enforce Primary Key Constraint toggle in the Table Editor if the source of your table or MV has duplicate data that you do not want to bring to Incorta. Disabling this option enhances the load time and performance and memory requirements during full load jobs. However, this may cause data redundancy.
Incorta upgraded the jQuery library used from version 2.1.1 to version 3.6.0 to enhance its security.
In addition to the new features and major enhancements mentioned above, this release introduces some additional enhancements and fixes that help to make Incorta more stable, engaging, and reliable.
- Fixed an issue with rejected rows due to data type mismatch in physical schema tables that use CSV files as a data source, in which the number of rejected rows was different from the actual ones in the downloaded CSV file
- Fixed an issue that caused a warning requiring a full load of schema not to appear when editing a function or encrypted column
- Resolved an issue where schema tables with a trailing space failed to load
- Resolved an issue for certain tables that caused requesting incorrect load type, especially, a full load rather than a staging load
- Fixed an issue in which selecting a deleted object in the load status menu would redirect the user to an incorrect page and would not display an error message denoting that the object is missing
- Fixed an issue in which the Save without discovery option was erroneously available for Incorta SQL tables
- Fixed an issue where a user could erroneously select to add Incorta SQL table data sources to multi-source tables
- Resolved an issue in which an encrypted measure or dimension column would not have its encryption status set to false when the column’s function was changed to a key
- Resolved an issue that caused loading Incorta SQL and Analyzer tables to fail with “Snapshot tables do not support full load” error
- Fixed an issue where the geo column type would not display in the column type menu of a table
- Resolved an issue where a table using drill-downs would display with all categories expanded
- Fixed an issue where schema loads involving the Data Agent could take longer than required to verify query execution
- Fixed an issue in which scheduled dashboard folders exported during the export tenant process were not imported with the tenant
- Fixed an issue with the Send Now option in which dashboards were not sent if the Pause Scheduled Jobs toggle was enabled in the Cluster Management Console (CMC)
- Fixed an inconsistency issue in the formula column data type between Business Schema View and Incorta Analyzer view
- Fixed an issue where the hint was not displayed if a user attempted to save a business view that referenced columns from multiple business views
- Resolved an issue where saving a listing table insight as a business schema view resulted in wrong display of the listed column names
- Enhanced the Aggregated Table visualization such that a column sort performed in the Analyzer is also reflected in the insight when downloaded to Excel or CSV
- Resolved an issue in the Aggregated Table visualization in which multi-row entries in the grouping dimension were trimmed
- Fixed an issue in which downloading and sending insights as PDF and HTML resulted in an error
- Fixed an issue in the Aggregated Table visualization in which conditional formatting did not appear for subtotals when the Merge Columns and Collapsed insight settings were enabled
- Resolved an issue where text in a right-to-left language overlapped on Organizational charts
- Fixed an issue in which the dashboard was not formatted correctly when emailed to an Outlook client
- Resolved an issue where updating a bookmark could result in an error
- Fixed an issue where sending data to a Google sheet could fail if a filter was applied to the dashboard
- Fixed an issue in which filter values may not appear in the UI when using the
- Fixed an issue that prevented pagination in SQL-view-based insights
- Resolved an issue where an error could occur in rendering an insight when applying a filter
- Enhanced the performance of joining rows between base tables and target tables
- Fixed an issue where enabling Merge Columns in the Analyzer caused an isight rendering error
- Resolved an issue where column data would not display when the last column in the data source query had an alias name
- Enhanced the logs by adding the column name to thread names
- Resolved an issue in which running notebook resulted in an error when cluster mode was enabled
- Resolved an issue where in some cases SparkSQL queries on a Notebook could fail
- Fixed an issue where auto-complete for spark Python in a Notebook would not suggest the correct function
- Resolved an issue where fetch size settings were not respected when using an ODBC driver in certain configurations
- Resolved an issue in which saving Notebooks with two paragraphs in MVs resulted in loss of paragraphs
- Resolved an issue where a backed-up tenant would not properly be restored
- Fixed an issue where restoring and overwriting a tenant could result in only deleting the tenant
- Resolved an issue where the path of a restored tenant would not display in the tenant details
- Resolved an issue in which the CMC logs were not being saved
- Fixed an issue that caused importing old tenants to fail
- Resolved multiple Japanese translation issues in the Content Manager and the Scheduler
- Resolved an Incorta Metadata error when loading the SQL_Audit table
- Fixed an issue that caused Incorta to fail to access a tenant created on an Azure Data Lake Storage (ADLS) that existed on the Azure China cloud
- Fixed an issue with external session variables that referenced an SQL Server data source, which caused testing the query to fail
- Resolved an issue where guide videos in the load data window were not available
- Fixed an issue in which an Oracle metadata DB tenant could not be imported to a MySQL metadata DB
- Fixed an issue where some median formulas could cause an unnecessary increase in the log file size
- Enhanced the security by upgrading the Apache Log4j2 used in different components to the latest version 2.17
After upgrading to the 2022.4.0 release, all existing global variables referenced in physical schema tables or MVs will function as expected. However, if you try to test these variables, it will throw errors. In addition, you must use a
queryDistinct expression when updating any of the existing global variables.
The following is a list of the known issues in this release, with a workaround if it is available:
- No validation on functions is triggered in the Formula Builder when creating global variables.
- Global Variables will not appear on the list of variables when referenced in a column individual filter, filter option, or applied filter, that is when you type
$in the search values box. However, you can add the global variable manually, if applicable.
Starts Withare examples of functions that accept a global variable as a filter value to be added manually.
- Global Variables that return a list of values are not functioning properly when referenced in MVs or individual filters
- Text in right-to-left languages is aligned left on Organizational charts. However, it is read normally from right to left.
- Changing the script language used for a loaded MV from Spark SQL to Incorta PostgreSQL throws errors
- Using the
DISTINCToperator on a measure in a Pivot table insight that is built over an Incorta SQL View throws errors
- Exporting a tenant, from a 2022.4.0 cluster, with a global variable that uses the newly supported syntax (
queryDistinctfunctions), and then importing it to the same or another 2022.4.0 cluster will cause the Edit Global Variable dialog to show an empty Value. You can manually edit the value to enter the query expression again.
- Incremental load jobs of tables with SQL-based data sources, such as Microsoft SQL Server and Oracle, will fail if the Incremental Script has multiple conditions with the
?option to retrieve new and updated data.
- Formula columns that reference global variables are evaluated as string in the following cases:
- Importing a tenant
- Restoring a tenant
- Starting up a server or a cluster
- Global variables with date or timestamp data types are not evaluated properly when referenced in the Analyzer.
- After changing the data source of an object in an imported physical schema or tenant to an Incorta SQL Table or changing an Incorta SQL Table in an imported physical schema or tenant to another object type, MV for example, the Disable Full Load option will be automatically enabled, which will lead to skipping these objects in the next schema full load jobs.
- You need to either disable the feature manually for objects that support disabling full loads (physical schema tables and MVs with incremental load enabled) or recreate the objects.
- Fixed an issue that caused changing the order of columns in a loaded physical schema object not to be correctly reflected in the Analyzer
- Fixed an issue where applying conditional formatting to a measure column in a tabular insight with dynamic hierarchy caused some values in the measure column not to appear in the Analyzer
- Fixed an issue where using the Search box in the dashboard resulted in an error
- Fixed an issue that caused the Fixed Placement setting option not to properly function
- Fixed a case while importing a tenant where the Next button was available although the Path field was empty
- Fixed a UI issue that caused variables and functions to appear in blue in the Functions and Variables panel of the Formula Builder