Release Notes Incorta 2024.7.5

Release Highlights

Building on the previous 2024.7.x releases, the Incorta 2024.7.5 maintenance pack introduces significant new features and numerous enhancements, further optimizing performance and improving the user experience.

Here are some of the key highlights in this release:

• Expanded data delivery with Snowflake as a schema data destination
• Enhanced visualizations, including advanced Y-axis scaling, granular control over dynamic group-by dimensions, and intuitive date and timestamp filtering
• Improved Advanced SQLi with better native Hive and Spark SQL connectivity and automatic Metastore synchronization
• Support for anomaly detection in Copilot and intuitive responses, defaulting to native Incorta charts with a detailed summary
• An Accelerated Engine startup and an improved mechanism to force load columns
• Additional Inspector reports for deeper environmental insights

Upgrade Considerations

CentOS Linux 7.x deprecation

Starting with 2024.7.5, support for the CentOS Linux 7.x operating system is officially discontinued. Customers currently running Incorta on CentOS Linux 7.x must upgrade their underlying operating system to a supported version before upgrading to Incorta 2024.7.5.

Installation process updates

Starting with this release, the installation process for Incorta will run exclusively in console mode when using the install script, and the GUI mode will no longer be available. Just run ./install to start the installation or upgrade process.

Clusters with Copilot-Slack integration

Please contact the Support team to plan for upgrading your 2024.7.4 cluster if it has active integration between Incorta Copilot and Slack.

Lineage information availability

Due to background processing of dependency calculations, lineage information may be temporarily unavailable if requested immediately after the Analytics Engine startup. In such cases, a message will be displayed indicating that lineage details are not yet accessible.

Data Agent considerations

The 2024.7.5 release uses the Data Agent version 9.2.3. Please upgrade to the required version.

What’s new

• Snowflake as a schema data destination
• Dashboard and Visualization enhancements
  • Advanced scaling options for the Y-axis
  • Limit dynamic-group-by dimension selections
  • Enhanced experience for filtering date and timestamp columns
  • Streamlined downloads: One request at a time
• Advanced SQLi enhancements
  • Improved Performance using Hive and Spark SQL
  • Spark Metastore sync enhancements
• Copilot enhancements
  • Defaulting to Incorta-native charts
  • Anomaly detection
• Accelerated Engine startup
• Improved Force Reload Column Mechanism
• A new SDK component: Multi-slicer
• Inspector Tool enhancements
  • duplicateDashboardHiddenInsight report
  • duplicateSchemaNames report
  • scheduleJobs report

Snowflake as a schema data destination

This release marks a significant enhancement to Incorta's data delivery functionalities. Complementing existing support for BigQuery and Synapse, Snowflake has been added as a new schema data destination. This integration empowers schema managers and administrators to seamlessly send processed data to Snowflake.

For details about the Snowflake data destination prerequisites and configurations, refer to Concepts → Data Destination → Snowflake schema destination properties.

Dashboard and Visualization enhancements

This release introduces multiple improvements to dashboards and visualizations, including the following:

• Advanced scaling options for the Y-axis
• Limit dynamic-group-by dimension selections
• Enhanced experience for filtering date and timestamp columns
• Streamlined downloads: One request at a time

Advanced scaling options for the Y-axis

Advanced y-axis scaling options are now available in chart visualizations, enabling dashboard developers to better represent numeric data, especially in scenarios involving skewed datasets, wide value ranges, or zero or negative values. These new options provide greater flexibility and precision, ensuring clearer and more insightful visualizations. The careful selection of a scaling option is crucial, as it profoundly impacts how viewers perceive data relationships and trends.
The Scaling property is now available as an insight setting and includes the following options:

• Default (linear):: The standard scale, where values are mapped proportionally
• Square Root: Compresses large positive values and emphasizes small values; useful for positively skewed data
• Biquadratic: Applies stronger compression to large positive values than Square Root
• Logarithmic: Displays data spanning orders of magnitude, emphasizing relative changes rather than absolute differences
• SymLog (Symmetric Logarithmic): Logarithmic with a linear region around 0; useful for data with both negative and positive extremes

Limit dynamic-group-by dimension selections

This release provides dashboard developers with granular control over dynamic-group-by dimensions, improving performance and ensuring system responsiveness.

At the insight level, within the Settings panel > General, you can now limit the number of dynamic-group-by dimensions a dashboard consumer can select. The Dynamic Group-by Default setting must be less than or equal to this newly configured limit. Dashboard consumers cannot exceed this limit when selecting from the Group By list, ensuring optimized query execution and a more focused analytical experience.

Enhanced experience for filtering date and timestamp columns

This release streamlines the filtering experience for dashboards and insights that utilize date or timestamp filter columns. Rather than manual input, a list of unique column values will be available to select from, ensuring consistent and intuitive filtering behavior across all data types.

Streamlined downloads: One request at a time

To optimize download performance and streamline the user experience, only one download request can be active at a time. When you start a download—whether for one or more tabs, a single insight from the Dashboard Manager, or a tabular insight within the Analyzer—all download buttons across the current and other dashboards and insights will be temporarily disabled. They'll remain inactive until your current download is complete or cancelled. This improvement helps reduce unnecessary load on the download service and enhances overall system stability.

Advanced SQLi enhancements

Improved Performance using Hive and Spark SQL

Major enhancements have been made to Advanced SQLi’s native Hive and Spark SQL connectivity, delivering significantly improved performance when running Advanced SQL queries. These improvements provide a faster, more efficient analytics experience and enhance integration with BI tools, such as connecting to Microsoft Fabric and Power BI (import mode) via the Spark SQL connector, and to tools like DBVisualizer via the Hive connector.

Spark Metastore sync enhancements

Spark Metastore now automatically syncs when physical or business schemas are created using the create or import options. This ensures consistent metadata availability beyond the previously supported triggers: Schema Wizard and schema updates.

Copilot enhancements

Defaulting to Incorta-native charts

Starting with this release, Incorta defaults to generating native insights when submitting questions to Copilot—no need to prepend /insight to your inquiry. The answer will also include:

• A detailed summary of the results
• A downloadable CSV file
• A tabular view as an alternative to the visual chart

Copy
{
  "general": {
    "clients_default_shortcut": {  
      "dashboards_v2": "chat"
    }
  }
}

Anomaly detection: Identifying outliers

Using the new functionality, /detect-anomalies, Copilot can now identify unusual patterns or outliers in data that deviate significantly from expected behavior. For example, Copilot can group daily or weekly sales figures to identify "normal" patterns. Copilot then highlights outlier dates with unusually high or low sales.

Accelerated Engine startup

This release significantly accelerates Analytics Engine startup time, making it available to receive queries much sooner. Key processes like dependency calculation and lineage warnings now run asynchronously in the background, resulting in a more responsive and efficient startup experience.

This feature is enabled by default. To disable it:

• For On-Premises environments, in the Analytics service.properties file, set the engine.dependency.component.update.in.background.enabled to false, then restart all Analytics services.
• For Cloud environments, contact the Support team.

Improved Force Reload Column mechanism

This release improves the Force Reload Column mechanism by limiting the write lock to be used only for updating metadata, evicting columns, and reading increments, while reloading columns uses a read lock, allowing for better concurrency during sync operations. This enhancement will result in an error only when opening the dashboard during the exclusive write lock, while the column or insight will appear as loading during column reading.

A new SDK component: Multi-slicer

This release introduces the new Multi-slicer component, designed to enhance user experience by consolidating multiple dropdown-type slicers within a single component. The Multi-slicer component allows you to make all selections across various filtering fields before applying them with a single Apply action.

The Multi-slicer component supports vertical and horizontal arrangements, with an option to show field labels above each dropdown list.

Inspector Tool enhancements

The Inspector Tool now generates additional reports as part of its output. You can modify the InspectorMetadata schema and create new tables based on the new CSV files.

• duplicateDashboardHiddenInsight
• duplicateSchemaNames
• scheduleJobs

duplicateDashboardHiddenInsight report

The duplicateDashboardHiddenInsight report identifies dashboards with insights built on result sets that share the same ID.

duplicateSchemaNames report

To help maintain consistent naming conventions and prevent potential ambiguities, the duplicateSchemaNames report lists physical and business schemas that share the same name, yet are distinguished only by their letter case (for example, SalesData and salesdata).

scheduleJobs report

The scheduleJobs report offers detailed information about scheduled jobs, such as load plans, sent dashboards, and data alerts. For each scheduled job, the report shows details, including the job name, type, status, frequency, and owner.

Enhancements and Fixes

Enhancements

Fixed Issues

Known Issues

For all of the known issues and workarounds in Incorta’s latest releases, refer to Known Issues.