Guides → Upgrade to Latest On-Premises Releases

Prior to upgrade, please review the Release Model. The Release Model describes the various release statuses: Preview and Generally Available. Refer to the Release Model to identify the release version and status that is most suitable for upgrading your Sandbox, Developer, User Acceptance Testing (UAT), or Production environment.

Upgrade Considerations

The following are the upgrade considerations per On-Premises release starting 2024.1.3.

2024.1.3

Considerations for MySQL 8

Before upgrading clusters that use a MySQL 8 metadata database from a 6.0.x or earlier release to 2024.1.x or 2024.7.1, execute the following against the Incorta metadata database:

Copy
ALTER TABLE `NOTIFICATION` MODIFY COLUMN `EXPIRATION_DATE` TIMESTAMP NULL DEFAULT NULL;

UPDATE `NOTIFICATION` SET `EXPIRATION_DATE` = NULL WHERE CAST(`EXPIRATION_DATE` AS CHAR(19)) = '0000-00-00 00:00:00';
COMMIT;

Spark 3.x

Spark 3.x requires a tmp directory with read, write, and execute permissions. You need also to specify this directory in the CMC and the spark-env.sh file.

1. Create a new tmp directory with the required permissions or use the <InstallationPath>/IncortaNode/spark/tmp directory.
2. Add the following configurations to this file <InstallationPath>/IncortaNode/spark/conf/spark-env.sh:

   Copy
   SPARK_WORKER_OPTS="-Djava.io.tmpdir=<DirWithPremissions>"
   SPARK_LOCAL_DIRS=<DirWithPremissions>
   
   

3. In the CMC > Server Configurations > Spark Integration > Extra options for Materialized views and notebooks, add the following options:

   Copy
   spark.driver.extraJavaOptions=-Djava.io.tmpdir=<DirWithPremissions>;spark.executor.extraJavaOptions=-Djava.io.tmpdir=<DirWithPremissions>
   
   

4. In the CMC > Server Configurations > Spark Integration > SQL App extra options, add the following options:

   Copy
   spark.driver.extraJavaOptions=-Djava.io.tmpdir=<DirWithPremissions> -Dorg.xerial.snappy.tempdir=<DirWithPremissions>;spark.executor.extraJavaOptions=-Djava.io.tmpdir=<DirWithPremissions> -Dorg.xerial.snappy.tempdir=<DirWithPremissions>
   
   

Next-generation loader

With the introduction of the new-generation loader, Incorta automatically detects inter-object dependencies within a load plan during the Planning phase of a load job. The Loader Service utilizes these detected dependencies and the user-defined load order within the schema to create an execution plan for loading objects. However, it’s important to note that using both the MVs' user-defined load order and automatically detected dependencies may result in an execution plan with cyclic dependencies, leading to load job failures. To avoid such failures, it is recommended to delete the MVs' user-defined load order before upgrading from a release before 6.0.

Connectors Marketplace

Starting 2024.1.3 On-Premises release, Incorta has introduced the Connectors Marketplace, allowing administrators to install, upgrade, and downgrade connectors independently from any Incorta release.

• After upgrading to 2024.1.3, the connectors’ files must be placed under the /marketplace/connectors/ directory. If this directory does not exist, create it under the system tenant path, which may vary according to Incorta installation. Afterward, you must unzip the ext_connectors.zip file that exists in the Incorta installation package and copy all the connectors’ folders from the unzipped directory to /marketplace/connectors/.
• Incorta moved the custom CData connectors from <IncortaNode>/extensions/connectors/customCData/ to <IncortaNode>/extensions/connectors/shared-libs/.
• For custom SQL connectors, you must move your custom SQL JDBC driver jars from <IncortaNode>/runtime/lib/ to <IncortaNode>/extensions/connectors/shared-libs/. In case of using a Data Agent, you must also copy these jars to incorta.dataagent/extensions/connectors/shared-libs/ on the Data Agent host.
• In case you need additional jars to use with the Oracle connector to support advanced options like XML, you must add or move the necessary libraries from <IncortaNode>/runtime/lib/ to <IncortaNode>/extensions/connectors/shared-libs/sql-oracle/.

Considerations for the MySQL driver

Starting 2024.1.3, the MySQL driver will no longer be included in the Incorta installation package or the Data Agent package. You must provide your own MySQL driver.

• For details about how to deploy your MySQL driver for Incorta installations, refer to Guides → Install MySQL Driver.
• For details about how to deploy your MySQL driver for Data Agent installations, refer to Tools → Data Agent.

Metadata database check

Upgrading to the Latest releases requires upgrading the metadata database to support multi-group load plans and migrate existing schema load jobs or load plans. Before upgrading to 2024.1.x or 2024.7.x, contact Incorta Support to perform a database check to inspect and detect any issues with your metadata database that might cause the metadata database upgrade to fail. If issues are detected, Incorta Support will run scripts against the metadata database to delete the records causing these issues.

SAP connector JCo files

Starting 2024.1.3, the SAP ERP Connector will stop bundling the JCo 3 libraries. For more information, refer to License Terms for SAP Connectors. Accordingly, you must manually get the needed libraries from SAP Java Connector and place them under <INCORTA_HOME>/IncortaNode/extensions/connectors/shared-libs/sapjco3.

Public API v2

The response of the /schema/{schemaName}/list endpoint has been updated. Some parameters have been renamed while additional parameters are now available to display error messages, the source column of runtime business view columns, and their data types. For more details, refer to Public API v2 → List Schema Objects.

Columns with Integer data type

In previous releases, Incorta wrote Integer columns in Parquet as Long. Starting 2024.1.3, Incorta writes Integer columns in Parquet as Integer for all newly created tables. For previously created tables, Incorta converts Integer columns written as Long in Parquet files to Integer during full load jobs, while Incorta keeps these columns unchanged during incremental loads.

As a result, after performing a full load of a table with Integer columns, it is recommended that you perform a full load of its dependent schema objects to ensure data consistency.

To migrate all Parquet files to have Integer data without fully loading your objects, the administrator can turn on the Enable Parquet Migration at Staging option in the Cluster Management Console (CMC) > Server Configurations > Tuning and perform a load from staging for all your objects with Integer columns.

Notes:

• Turning the Enable Parquet Migration at Staging option on adds a new step that runs on Spark to the staging load jobs. Ensure Spark has sufficient resources to migrate the required Parquet files.
• The migration of Parquet files during staging load occurs only once per object.
• Load tables before loading MVs that read from these tables.
• When loading an object fully, you must load dependent objects or load all objects from staging after turning the Enable Parquet Migration at Staging option on.
• The new behavior might affect existing scripts referencing Integer or Long data, whether columns, values, or variables.

Schema names

Starting 2024.1.3, the names of newly created schemas and business schemas are not case-sensitive. Therefore, you cannot create a new schema or business schema that shares the same name with existing schemas or business schemas using different letter cases. To maintain backward compatibility, the names of existing schemas and business schemas will not be affected.

Caching mechanism enhancements

The caching mechanism in Analyzer views for dashboards has been enhanced by caching all columns in the view to prevent query inconsistencies. To optimize performance and reduce off-heap memory usage, creating views with only the essential columns used in your dashboards is recommended.

Like function fix

As a result of fixing an issue with the like() function, you must load all tables that use this function from staging if you are upgrading from releases before 6.0.2.

External Notebooks

In the case of using external Notebooks, you must force reinstalling the Python library after upgrading from a release before 6.0.

SDK Component installation

For now, SDK Component installation is only supported on clusters that use a MySQL metadata database.

2024.1.4

SQLi service consideration

Starting 2024.1.4, the SQLi service must be started before creating or processing SQL views over non-optimized tables or Incorta-over-Incorta tables via Spark port; otherwise, Incorta throws an error.

2024.1.5

Single Logout for SAML2 SSO providers

Starting 2024.1.5, Incorta supports Single Logout (SLO) for SAML2 SSO identity providers, including Azure Active Directory (AD) and OneLogin. When users sign out from Incorta, they are automatically signed out from their SAML2 SSO identity providers.

The Single Logout URL on both Incorta and the identity provider must be set to Incorta’s login page URL, for example, https://myCluster.cloud.incorta.com/incorta/!myTenant/ or https://10.10.1.5:8080/incorta/!myTenant/.

In the case of OneLogin, set the SAML initiator option to Service Provider.

Update the SSO configurations on Incorta and add the following settings:

• onelogin.saml2.idp.single_logout_service.url: The logout URL provided in your SSO provider's metadata.xml
• onelogin.saml2.sp.single_logout_service.url: Incorta’s login URL, for example, https://myCluster.cloud.incorta.com/incorta/!myTenant/ or https://10.10.1.5:8080/incorta/!myTenant/
• onelogin.saml2.sp.single_logout_service.response.url: Incorta’s login URL

2024.1.7

After upgrading to 2024.1.7 and with the introduction of Incorta Premium, the following features will be impacted:

• Notebook for business users: Enable Incorta Premium to be able to access existing business notebooks and create new ones.
• Data Studio: To avail the Data Studio tab in the Analytics platform, enable Incorta Premium, enable Data Studio for Cloud clusters in the Cloud Admin Portal > Configurations, and enable and configure it in the CMC > Server Configurations > Incorta Data Studio.

2024.7.1

JDK supported versions

JDK 8u144 is no longer supported, and you must upgrade to JDK 8u171 or later.

Spark 3.4.1

Starting this release, Incorta uses Spark 3.4.1. All Incorta applications, including the Advanced SQL Interface, will use one unified Spark instance (<incorta_home>/IncortaNode/spark/) to handle the different requests. For more details about the difference between the Spark versions, refer to the 3.4.1 Migration Guide.

As a result of using a unified Spark instance, you must consider the following when upgrading On-Premises clusters that have the Advanced SQLi previously enabled and configured:

• Take into consideration the resources required by the Advanced SQLi when configuring the Spark worker resources in the CMC.
• Post upgrade, make sure that the CMC is started and running before starting the services related to Advanced SQLi, such as the Advanced SQL service and Kyuubi, to have the configuration values updated to refer to the unified Spark instance instead of SparkX.

Deprecating Incorta custom Delta Lake reader

Starting this release, Incorta has deprecated its custom Delta Lake reader and started to use only Spark Delta Lake reader to read Delta Lake files.

Load jobs and scheduled plans

Before starting the upgrade process, do the following:

• For each tenant in your cluster, pause the load plan schedulers: In the Cluster Management Console (CMC) > Tenant Configurations > Data Management, turn on the Pause Load Plans toggle.
• Stop all load jobs, including in-progress and in-queue jobs: In the Schema Manager, select Load Jobs, and then select Stop for each job you want to stop.

Load filters

During the upgrade, existing load filters for a table will be migrated as partial loading filters. After the upgrade, tables with load filters will require a load from staging.

Accounting for nulls in the Loader Service

Starting 2024.7.1, Incorta Loader accounts for null values in the following contexts whether or not you enable Null Handling in the CMC:

• Deduplication and PK-index calculations:
  The Loader Service will consider null values as undefined values. Thus, null values are no longer considered zeros or empty values. When loading data in full or incremental load jobs, the Loader Service will retrieve or update records with null key values apart from records with zero or empty key values. However, to preserve backward compatibility, Incorta considers a null value equal to another null value when retrieving or updating data. Accordingly, only one record with a null value will be loaded or updated if the Enforce Primary Key Constraint option is enabled.

  
  

• Join calculations and join filters:
  The Loader Service will account for records with null values individually. During join calculations, a null value does not equal another null value, zero, or empty value. Additionally, the Loader Service will account for null values when evaluating the join filter.

Intelligent Ingest

After upgrading to 2024.7.1, previously configured incremental ingest jobs may fail or cause data inconsistencies at the destination. To prevent these issues, perform a full ingest before running the first incremental ingest post-upgrade.

Note: This requirement does not apply to schemas configured for ingestion for the first time after the upgrade as they will automatically undergo a full ingest during the initial ingest job.

Notebook for business users

Users who created business Notebooks in previous releases of Incorta will need the newly released Advanced Analyze User role for continued access in 2024.7.x.

Upgrades from 2024.1.7

A new version of the Excel Add-in has been introduced in 2024.1.7. This new version is supported in the 2024.7.x releases starting with 2024.7.4. Therefore, upgrading from 2024.1.7 to 2024.7.1, 2024.7.2, and 2024.7.3 is not recommended for the Excel Add-in users.

2024.7.2

Incorta Premium

After upgrading to 2024.7.2 and with the introduction of Incorta Premium, the following features will be impacted:

• Notebook for business users: Enable Incorta Premium to be able to access existing business notebooks and create new ones.
• Spark SQL Views: To create new, explore, edit, and visualize Spark SQL Views, enable Incorta Premium and Spark SQL View in the Server Configurations > Incorta Labs.
• Data Studio: To avail the Data Studio tab in the Analytics platform, enable Incorta Premium, enable Data Studio for Cloud clusters in the Cloud Admin Portal > Configurations, and enable and configure it in the CMC > Server Configurations > Incorta Data Studio.
• Copilot: To have access to Incorta Copilot and generative AI capabilities, enable Incorta Premium, enable and configure Copilot for Cloud clusters in the Cloud Admin Portal > Configurations, and enable and configure it in the CMC > Server Configurations > Incorta Copilot.

2024.7.3

Null handling

• Null Handling is now an independent feature, separate from Advanced SQL Interface (Advanced SQLi). You can enable or disable it as needed. After upgrading to 2024.7.3, clusters with Advanced SQLi enabled will automatically have Null Handling enabled as well.
• During join calculations, the Loader Service will handle null values based on the Null Handling CMC setting.
  • Disabled: The Loader Service treats Null values as zeros for numeric columns, empty strings for text columns, and empty dates for date columns.
  • Enabled: The Loader Service treats Null values as distinct values, not equivalent to zeros, empty strings, or other null values.

Spark Metastore

You must sync the Spark Metastore after upgrading to this release to ensure access to Spark SQL views and insights leveraging them. Go to the CMC > Clusters > <cluster_name> > Tenants > <tenant_name> > More Options (⋮), and then select Sync Spark Metastore.

2024.7.4

On-Premises clusters using MySQL

• For a smooth upgrade process for clusters with MySQL metadata database, ensure your metadata database server runs MySQL 8.
• Customers running Incorta On-Premises with a MySQL metadata database, where the MySQL server is configured to automatically add invisible primary keys to tables without one, should contact the Support Team for assistance in upgrading their clusters to this release.

Null Handling updates

• In this release, the Incorta Engine will treat null values in numeric columns as zeros and data columns as the maximum Java date when performing arithmetic operations (+, -, *, /, %), regardless of the Null Handling setting in the CMC.

  • This change is limited exclusively to arithmetic operations and does not affect other functions, such as sum, average, addMonths, or similar calculations.
  • Formulas with decimal columns divided by null will throw an error instead of returning a null value.
  • However, when Null Handling is enabled, the Advanced SQL Interface will continue to treat null values as true nulls in arithmetic queries originating from third-party visualization tools, Spark SQL Views, and Business Notebooks.
    Important

    If Null Handling was enabled before the upgrade, you must reload schemas related to formulas containing null values with arithmetic operations from staging to ensure consistent data.

• This release has fixed the discrepancy where the null category (coloring dimension) was not displayed in charts when using numerical or date grouping dimensions while it was displayed for string grouping dimensions. After upgrading to this release, all charts with a coloring dimension that contains null values will, by default, display a null category. To hide it, apply a Not Null filter to the coloring dimension column.

Caching updates

• The default value of the Maximum Cached Entries has been increased from 2,000 to 20,000. This change will not impact values manually set before the upgrade.
• A new option is available to control the maximum size of insight query results to be cached. The default is 30 MB.

Engine Audit updates

In this release, a new version of the Enhanced Engine Audit, engine_audit.2.1, is now available with new audit files and updates to existing ones.
After upgrading, audit data will be stored in the engine_audit.2.1 folder. If you have schemas reading from the old engine_audit.2.0 folder, create multi-source tables that read from both folders to maintain a complete audit history.

Clusters on Azure

To upgrade your clusters deployed on Microsoft Azure to 2024.7.4, contact the Support Team.

Detailed upgrade guides

Here are the guides detailing how to upgrade to the latest On-Premises releases starting 2024.1.3:

• Guides → Upgrade from 5.0.x to Latest Releases
• Guides → Upgrade from 5.1.x or 5.2.x to Latest Releases
• Guides → Upgrade from 6.0 to Latest Releases