Release Notes Incorta 2024.7.4

Release Highlights

2024.7.4 builds on the 2024.7 release with new product capabilities and several enhancements to existing features.

Here are some of the key highlights:

• Introducing the Incorta data catalog for data governance and cataloging active metadata across all Incorta data assets
• Insights over SQL result sets
• Advanced analytical capabilities within the Copilot
• Incorta Copilot integration with Slack
• Additional usability enhancements in Incorta copilot
• Null handling enhancements
• Caching and query performance enhancements
• Securing sensitive data in core-site.xml in ADLS Gen2 tenants

Upgrade Considerations

On-Premises clusters using MySQL

• For a smooth upgrade process for clusters with MySQL metadata database, ensure your metadata database server runs MySQL 8.
• Customers running Incorta On-Premises with a MySQL metadata database, where the MySQL server is configured to automatically add invisible primary keys to tables without one, should contact the Support Team for assistance in upgrading their clusters to this release.

Null Handling updates

• In this release, the Incorta Engine will treat null values in numeric columns as zeros and data columns as the maximum Java date when performing arithmetic operations (+, -, *, /, %), regardless of the Null Handling setting in the CMC.

  • This change is limited exclusively to arithmetic operations and does not affect other functions, such as sum, average, addMonths, or similar calculations.
  • Formulas with decimal columns divided by null will throw an error instead of returning a null value.
  • However, when Null Handling is enabled, the Advanced SQL Interface will continue to treat null values as true nulls in arithmetic queries originating from third-party visualization tools, Spark SQL Views, and Business Notebooks.
    Important

    If Null Handling was enabled before the upgrade, you must reload schemas related to formulas containing null values with arithmetic operations from staging to ensure consistent data.

• After upgrading to this release, all charts with a coloring dimension that contains null values will, by default, display a null category. To hide it, apply a Not Null filter to the coloring dimension column. See Fixed Issues for details.

For more details, refer to References → Null Handling.

Caching updates

• The default value of the Maximum Cached Entries has been increased from 2,000 to 20,000. This change will not impact values manually set before the upgrade.
• A new option is available to control the maximum size of insight query results to be cached. The default is 30 MB.

For more details, see Caching Enhancements.

Engine Audit updates

In this release, a new version of the Enhanced Engine Audit, engine_audit.2.1, is now available with new audit files and updates to existing ones.
After upgrading, audit data will be stored in the engine_audit.2.1 folder. If you have schemas reading from the old engine_audit.2.0 folder, create multi-source tables that read from both folders to maintain a complete audit history.

Clusters on Azure

To upgrade your clusters deployed on Microsoft Azure to this release, please contact the Support Team.

Data Agent Considerations

The 2024.7.4 release uses the Data Agent version 9.2.2. Please upgrade to the required version.

New Features

Dashboards, Visualizations, and Analytics

• Insight over SQL queries
• Customizable null representation in tables
• Dashboard link sharing with Mobile users

Data Management Layer

• Data Catalog and Classification

Incorta Copilot

• Enhanced user experience (UX)
• Support for advanced analytics features
  • Key influencers
  • Clustering
  • Forecasting
• Customizable response detail level
• Integration between Copilot and Slack
• AI-powered insight descriptions with Incorta Copilot
• Copilot audit enhancements with a new data app

Architecture and Application Layer

• Caching enhancements
  • Updates to the caching CMC options
  • Control the size of query results cached in memory
• Enhanced dashboard performance with shared filter caching
• Optimized query performance with parallel rollup
• Secure sensitive data in core-site.xml

Integrations

• A new Excel Add-in version: 16.9.0

Dashboards, Visualizations, and Analytics

Insight over SQL queries

You can now create insights based on Spark SQL result sets via the Analyzer or through natural query generation using Incorta Copilot (Nexus).

• In the Analyzer, create a result set based on a Spark SQL query using the new option Via Spark SQL. Then, use the result set to build an insight.
• With Incorta Copilot (Nexus), start your question with /insight to generate a SQL-based answer in the form of an Incorta-native insight. You can further refine the generated insight and its result in the Analyzer or directly add the SQL-based insight to a new or existing dashboard.

Customizable null representation in tables

In this release, CMC and cloud admins can specify how null values are displayed in tabular insights when previewing them in Incorta and when sharing or exporting them in PDF, JPEG, SVG, or HTML formats. Navigate to the CMC > Tenant Configurations > Customizations, and then for the Null Value Representation setting, select one of the available options: null, empty, and dash.

Dashboard link sharing with Mobile users

This release improves the experience of sharing dashboard links with authorized Mobile users, providing a seamless and user-friendly approach.

• The link is available in text format as well as a QR code for easy sharing.
• Clicking the link or scanning the QR code automatically launches the Incorta Mobile App for quick access.
• The link includes all necessary details to open the dashboard, such as the tenant name and dashboard GUID.

Data Management Layer

Data Catalog and Classification

In this release, Incorta introduces a new data catalog and classification solution that provides advanced data discovery and classification capabilities. This solution empowers businesses to maximize data value while ensuring accuracy, quality, regulatory compliance, and usability.

Key features and capabilities:

• Data Catalog – Comprehensive discovery and metadata management for data assets
• Data Classification – Built-in classification for sensitive, confidential, and personal data

Data Catalog

The Data Catalog is a centralized data inventory that helps users efficiently discover, understand, and manage their data assets, including data sources, physical schemas, business schemas, tables, views, columns, and dashboards.

Once the Data Catalog is enabled, it is automatically populated by scanning the tenant’s metadata. You can then:

• Create and assign domains, glossary terms, and tags to data assets.
• Add documentation to assets, domains, and glossary terms for better data context.

Data Classification

The Data Catalog includes predefined classifications for identifying sensitive, confidential, and personal data. Additionally, classified data can be masked for unauthorized users, enhancing security and compliance.

Incorta Copilot

Enhanced user experience (UX)

This release introduces an improved user experience when dealing with Incorta Copilot.

• New shortcuts to get column names and data values
• Autocomplete for column names and data values
• Semantic search across all relevant datasets without specifying a specific business view as context.

Support for advanced analytics features

This release introduces advanced analytics capabilities in Incorta Copilot, enabling you to gain deeper insights and confidently make data-driven decisions. The new features help you uncover key patterns, identify influential factors, and predict future outcomes to enhance business performance.

Key ML Features:

• Key Influencers – Identify the factors that drive and impact a given metric.
• Clustering – Automatically segment data into meaningful groups.
• Forecasting – Predict future business trends based on historical data.

Key Influencers: Understand what drives your data

Copilot helps you discover the most influential factors affecting a particular metric, allowing for a better understanding of data trends and more informed decision-making.

Example: Identify key drivers of customer churn, such as pricing changes or product issues, and take proactive steps to improve retention.

There are two key influencer methods:

• Explain Analysis – Provides a high-level summary of the factors influencing your business by analyzing all measures and dimensions in the dataset.
• Tree Explain Analysis – Generates an interactive tree diagram, revealing how different factors contribute to a specific outcome. Expand the tree to explore each factor in detail and uncover deeper insights.

Clustering: Automatic data segmentation

Copilot can automatically group data into meaningful segments based on shared characteristics, helping you identify trends and optimize decision-making.

Example: Group customers with similar buying habits to create targeted marketing campaigns or analyze frequently bought-together products to enhance cross-selling strategies.

Forecasting: Predict future outcomes with confidence

With its built-in forecasting capabilities, Copilot can predict future business trends based on historical data, enabling improved strategic planning.

Example: Forecast future sales revenue to optimize inventory management, production planning, and sales strategies.

Customizable response detail level

This release introduces two additional configurations in Incorta Copilot’s settings: Summary and Verbosity, giving users greater control over AI-generated responses, performance, and data security

• Summary: Enables or disables the display of summarized AI-generated responses to asked questions. This option is enabled by default.
• Verbosity: Controls the level of detail in generated summaries. You can choose between Detailed or Brief summaries. This option is available only when Summary is enabled, with Detailed set as the default.

Integration between Copilot and Slack

Starting with this release, you can now integrate Incorta Copilot with Slack. This powerful integration directly provides real-time, data-driven insights to your Slack workspace, empowering your team to make smarter decisions, enhance productivity, and promote collaboration without leaving Slack, ensuring the platform's security and governance are maintained. You can ask questions in plain English and receive instant responses in various formats, including summaries, CSV files, tables, and charts.

Prerequisites and requirements:
To get started, ensure you have the following:

• A Premium cluster
• Advanced SQL Interface enabled and configured
• Copilot enabled and configured
• The integration Manifest file
• Slack administrator or developer account (Recommended)
• A user with SuperRole or Copilot User role
• A valid personal access token (PAT)
• Verified business views enabled for Copilot

For more details, refer to Integrations → Incorta Copilot for Slack.

AI-powered insight descriptions with Incorta Copilot

In this release, you can use the generative AI capabilities of Incorta Copilot in the Analyzer to add descriptions to your insights automatically. These AI-generated descriptions are seamlessly populated into the Data Catalog and the Dashboard, enhancing data clarity and accessibility across the platform.

Copilot audit enhancements with a new data app

In this release, Incorta introduces an updated version of the Copilot audit files in the <instalationPath>/IncortaAnalytics/Tenants/<tenantName>/copilot_audits directory. This directory now includes:

• CopilotAudit.csv: Logs user session details and usage data, including queries, context, query responses, and statuses.
• CopilotFeedbackAudit.csv: Captures user feedback Copilot-generated responses.

Audit logging starts after users interact with Copilot by submitting a query or providing feedback on a response.

To streamline monitoring Copilot usage and feedback, Incorta has provided a new data app in the Marketplace. Simply install the data app, load data, and explore the logged details for improved monitoring and analysis.

Architecture and Application Layer

Caching enhancements

Updates to the caching CMC options

Incorta has applied the following updates to the caching options in the CMC > Tenant Configurations > Tuning:

• Maximum Cached Entries: Increased the default from 2,000 to 20,000. This change will not impact tenants where this value is manually updated before the upgrade.
• Maximum Cached Memory (%): Increased the maximum allowed percentage from 10% to 20%.
• Export to CSV/XLSX caching limit: Increased the maximum allowed size from 100 MB to 400 MB.

Control the size of query results cached in memory

The Query caching entry size limit for new tenant configurations is now available on the Tuning tab to specify the maximum size of insight query results that the Analytics Service caches in memory. The Analytics Service checks for a cached version before executing the insight query to enhance performance and minimize the resources required. You can select a value between 0 and 50 inclusive. The default is 30 MB.
The Analytics Service won’t cache insights in the following cases:

• Insights with a query result size that exceeds the Query caching entry size limit
• Disabling caching by setting the value of Query caching entry size limit or Maximum Cached Entries to 0
• Reaching the limit of the Maximum Cached Entries configuration

Enhanced dashboard performance with shared filter caching

This release introduces shared filter caching to improve the performance of rendering dashboards and insights when filters are applied.

Key enhancements:

• Shared filter caching – Caches dashboard-level filters, including prompts, applied filters, filter options, and presentation variables, in addition to runtime security filters.
• Optimized query execution – When insights on a dashboard share the same base table, cached filters enhance efficiency.
• Persistent caching for multi-user dashboards – Cached filters can be persisted in memory and shared across multiple dashboards and users, provided:
  • The dashboards use the same cache key.
  • The query meets the conditions in engine.properties, including cache entry size and total memory usage for long-term caching.

Limitations:

• Queries with the First Version or Last Version operator are not cached.
• Caches are deleted when any schema gets updated, or the total size is more than the configured value.
• A shared filter condition with more than 1000 values will not be cached.

Manage the feature:

The new feature is disabled by default. To enable this feature or customize its settings:

• For Cloud clusters, contact the Support Team.
• For On-Premises clusters, modify the engine.properties file in the Analytics Service by adding the following properties:

Optimized query performance with parallel rollup

This release significantly enhances query performance for datasets with large numbers of groups or dimensions by executing rollup calculations (that is, grouping dimension calculations) in parallel rather than sequentially, ensuring:

• Faster aggregations, optimized for queries exceeding 10 million groups, reducing execution time.
• Improved performance for complex analytical queries

This feature is enabled by default and available for both Cloud and On-Premises deployments. To disable it:

• For Cloud clusters, contact the Support Team.
• For On-Premises clusters, modify the engine.properties file in the Analytics Service directory and set the engine.concurrent_rollup_parallel_enabled option to false.

Secure sensitive data in core-site.xml

This release introduces a new method for securing sensitive credentials within the core-site.xml file. This method enables storing sensitive credentials within a Java KeyStore, enhancing data protection and improving security by preventing direct exposure of sensitive credentials in configuration files. This feature is now available for tenants using Azure Data Lake Storage (ADLS) Gen2.

For more details, refer to Guides → Configure a Tenant on ADLS Gen2 → Securing the core-site.xml using Incorta.

Integrations

Excel Add-in version 16.9.0

A new version of the Excel Add-in (16.9.0) is now available, featuring several enhancements and fixes:

• Enhanced User Experience: A simplified, more intuitive connection process to Incorta. You can now choose between Continue with direct login or Continue with SSO from the Login panel.
• Expanded Date Part options: Additional options for Date Part selection, beyond Full date, to accommodate diverse reporting needs.
• Resolved Issues:
  • Fixed an issue where numbers were truncated (rounded) in the Excel Add-in, ensuring data accuracy and numbers match the precision of those stored in Incorta.
  • Addressed a bug that caused errors when creating Pivot tables, improving reliability.

For more details, refer to Integrations → Excel Add-in.

Enhancements and Fixes

Enhancements

Fixed Issues

Known Issues

For all of the known issues and workarounds in Incorta’s latest releases, refer to Known Issues.