What's New in Incorta Cloud 2022.11.0

Release Highlights

Incorta introduces several improvements to data ingestion and data manipulation in this release. These improvements include a next-generation loader with a new architecture, a new connector for the Oracle Warehouse Management System (WMS), and support for fiscal calendars.

This release includes preview features, such as scheduled load plans with multiple schemas, scheduled dashboard delivery with bursting reports based on the recipient’s context, and extending parallel reading of columns into memory for all data types.

Additional features include support for Spark 3.3 on newly created clusters, enhancements to the Bubble visualization, and two new public API endpoints to load schemas and retrieve their details.

Note

The 2022.11.0 release is available for newly created clusters and new trial users. If you’re an existing customer upgrading from a prior release, please contact Incorta Support to schedule your upgrade.

Upgrade considerations

daysBetween function

In this release, the daysBetween function does not automatically round up the returned value to the nearest long value, instead, it returns and saves the value as double. To round the returned value and retain the old behavior, you have to wrap up the daysBetween function with the round or ceil function. To avoid errors that might occur after the upgrade when rendering insights and business schemas that reference columns with the daysBetween function, you have to load from staging all physical schemas where these columns exist.

Schemas with user-defined load order

With the introduction of the new-generation loader, Incorta automatically detects inter-object dependencies within a load plan during the Planning phase of a load job. The Loader Service utilizes these detected dependencies and the user-defined load order within the schema to create an execution plan for loading objects. However, it’s important to note that using both the MVs' user-defined load order and automatically detected dependencies may result in an execution plan with cyclic dependencies, leading to load job failures. To avoid such failures, it is recommended to delete the MVs' user-defined load order before upgrading to the 2022.11.0 release.

Next-generation loader

Incorta's next-generation loader comes with a new architecture that comprises multiple components. The new architecture effectuates the plan-based execution of load jobs by introducing a new Planning phase. During this phase, the Loader Service detects the dependencies among the objects in the load job, along with manually defined load groups, and creates a plan for loading objects accordingly. In addition, the Loader Service concurrently executes the extraction and transformation of objects in the load job in the same phase: the Running phase. The new architecture optimizes the Post-load calculations so that the Loader Service concurrently executes different calculation types. These enhancements eliminate previous limitations and open the door for more improvements.

The load job now goes through the following phases:

Planning: The Loader Service detects the dependencies among the physical schema objects in the load plan, checks manually defined load order groups, and accordingly creates a directed acyclic graph (DAG) plan to determine the object loading steps and order.
Running: The Loader Service concurrently executes the extraction and transformation (enrichment) of tables and materialized views (MVs) respectively, based on the DAG plan. Combining the extraction and transformation processes in one phase eliminates unnecessary waiting time.
- If the physical schema has manually defined load order groups where all tables are in a group and all MVs are in a different one, the Loader Service won’t execute the extraction and transformation in parallel.
Post-load: After completing the extraction and transformation of all tables and MVs in the load plan, the Loader Service performs Post-load calculations, such as formula columns, joins, and derived tables (Incorta SQL and Analyzer tables).
- The Loader Service no longer performs these calculations sequentially depending on the calculation type. Instead, it executes these calculations concurrently or sequentially according to their interdependency detected during the Planning phase.
- This phase now involves the Loading phase. For each calculation, the Loader Service loads into the engine memory only the columns that the service requires, completes the calculation, and then creates the required DDM files. The Loader Service then marks the unneeded columns so that they can be evicted from the memory when needed, which reduces the memory needed during this phase.

Note

The Schema Pool Size option in the Cluster Management Console (CMC) > Cluster Configurations > Tenant Configurations > Tuning is deprecated and replaced by the Maximum Concurrent Load Jobs option that controls the maximum number of load jobs that the Loader Service can handle at the same time.

Oracle WMS Cloud connector

The new Oracle Warehouse Management Software (Oracle WMS Cloud) application connector brings your logistics data into Incorta so you can start investigating your data and building insights. Oracle WMS Cloud is the logistics system in the Oracle Supply Chain Management and Manufacturing system.

To start using the Oracle WMS Cloud connector, do the following:

Log in to your Incorta Cluster.
Select the Data tab.
In the Action bar, select + New → Add Data Source.
Select the Oracle WMS connector under Application.
Fill in the following required information:

Property	Control	Description
Name your connector	text box	Enter a name for your data source
Domain	text box	Enter the URL for the Oracle WMS services API in the following form: `https://<SERVICE_NAME>.wms.ocs.oraclecloud.com/`
Username	text box	Enter the username for the data source
Password	text box	Enter the password for the data source
Pagination Mode	drop-down	Select the pagination mode required from the following available values: ● Paged ● Sequenced
Timeout in seconds	text box	Enter the time in seconds to wait for a response from the Oracle WMS server. The default is 30 seconds.
Max number of concurrent calls	text box	Enter the maximum number of concurrent calls that Incorta sends to the Oracle WMS server. The default is 5 calls.
Max number of requests per minute	text box	Enter the maximum number of requests done per minute to the Oracle WMS server. The default is 30 requests.

Test the connection by selecting Test Connection.
Select Ok.

Support for fiscal calendars

For customers with fiscal calendars that have a fiscal offset in months, Incorta now natively supports fiscal year calendars. This feature simplifies the setup and consumption of fiscal calendars in Incorta and allows for the rollup of transactional data into fiscal calendars. You can define the start of the fiscal year using the Fiscal Year option in the Cluster Management Console (CMC), under Cluster Configurations > Server Configurations > Customizations.

Date filters and date aggregations now honor fiscal calendars. In addition, multiple fiscal-calendar-based system variables, such as currentFiscalYearEnd, nextFiscalQuarterStart, and lastFiscalMonth, are available to select from when creating filters, formula columns, or queries. You can also apply different fiscal-calendar formats depending on the selected Date Part, such as short fiscal year (FYYY), fiscal year range (FYYY-YY), fiscal quarter (FQ#), and fiscal period (FMMM).

The following system variables are now available in Incorta to support fiscal years, quarters, and periods:

Fiscal year	Fiscal quarter	Fiscal month
currentFiscalYear	currentFiscalQuarter	currentFiscalMonth
currentFiscalYearStart	currentFiscalQuarterStart	currentFiscalMonthStart
currentFiscalYearEnd	currentFiscalQuarterEnd	currentFiscalMonthEnd
nextFiscalYear	nextFiscalQuarter	nextFiscalMonth
nextFiscalYearStart	nextFiscalQuarterStart	nextFiscalMonthStart
lastFiscalYear	lastFiscalQuarter	lastFiscalMonth
lastFiscalYearStart	lastFiscalQuarterStart	lastFiscalMonthStart

The following format options are available for fiscal date parts:

Date part	New format options	Examples
Fiscal year	● Long Year > `YYYY` ● Long Fiscal Year > `FYYYYY` ● Short Fiscal Year > `FYYY` ● Year Range > `YYYY-YY` ● Fiscal Year Range > `FYYY-YY`	● 2023 ● FY2023 ● FY23 ● 2022-23 ● FY22-23
Fiscal quarter	● No format > `#` ● Quarter > `Q#` ● Fiscal Quarter > `FQ#`	● 1 ● Q1 ● FQ1
Fiscal period	● No format > `#` ● Period > `M#` ● Fiscal Period > `FM#` ● Short Fiscal Period > `FMMM` ● Long Fiscal Period > `FMMMM`	● 1 ● M1 ● FM1 ● Apr ● April

You can use these new variables and date parts along with the existing ones to create comparisons between fiscal and Gregorian calendars. In addition, the ago and toDate functions now support the fiscal calendar. For example, by using the Fiscal Year and Fiscal Quarter date parts with the ago and toDate functions, you can compare fiscal-quarter-to-date data and year-ago-fiscal-quarter-to-date data without much effort.

Multi-schema load plans

As a Schema Manager, you can now schedule load plans that involve loading multiple physical schemas. This feature reduces the number of required load plans, expedites and facilitates data refresh cycles, and solves the dependency issues among schemas. The Loader Service calculates cross-schema joins for all schemas in the load plan only once, minimizing the total loading time for these schemas.

Note

Multi-schema loading is a preview feature in this release. To enable it on your cluster, contact Incorta Support.

This feature reduces the efforts needed to manage schemas, making it easier to load multiple interdependent schemas, if required.
You can start creating a multi-schema load plan either from the Load Plans tab under Scheduler or from Schema Manager.

You can specify a load type (full load, incremental, or staging) for each schema.
The Loader Service handles each load plan as one load job and automatically orders the steps (extraction, enriching, post-loading calculations, etc.) to deal with objects in the load job according to their dependencies.

Feature limitations and known issues

Load jobs with common schemas can’t run concurrently. As a result, you can’t start loading a schema that is part of a running load job, and the Scheduler can’t run a scheduled load plan that contains one or more schemas in a running load job.
Multi-schema load plans require enough memory to handle the concurrent loading of all schemas in the plan. If there’s not enough memory available on the cluster, load jobs will fail with an out-of-memory error.
For now, when one of the schemas fails in the Post-load stage, all schemas show the Finished with Error status even if other schemas load successfully.
The Loading Time for each schema in a multi-schema load plan shows the total elapsed time for the whole load job, not how long it takes each schema to load.
Stopping the loading of one of the schemas in a running load job stops the loading of all other schemas in the same load plan.
Adding a load plan from Schema Manager doesn’t display the total number of schemas correctly..

Bursting reports

Incorta now supports security context-based bursting reports where reports are distributed based on the recipients’ context instead of the sender’s context. For example, if salespeople in different regions each need a report showing the sales target for their country or region, use burst reports to send each salesperson only the information they need.

To burst a report, in the dashboard Send/Schedule a Report page, select the Burst Report check box. You can send or schedule a bursting report or dashboard for up to 300 internal users with Incorta accounts and at least view access rights to the dashboard. The report runs once but renders customized data based on the security filters or session variables of each recipient.

Note

Bursting reports is a preview feature. To enable this feature on your cluster, contact Incorta Support.

Bubble chart size enhancements

You can now set minimum and maximum percentage values for the radii of the bubbles within a bubble visualization. You can do that using the newly added property Radius in the pill properties of Size By tray.

Follow the next steps to be able to use the new property:

Log in to Incorta.
Select the Content tab.
From the Action bar, select + New → Add Dashboard.
Name your dashboard.
Select + Add Insight.
Choose Bubble from the Insight panel.
Add a column to the Size By tray.
Drag and drop the beginning and end of the Radius property to set the minimum and maximum values for the bubbles' radii.

Parallel column reading enhancements

Incorta now extends the parallel reading of columns into memory to include numeric and date data types. In addition, Incorta can store and use column stats of parquet files to optimize reading and loading compressed columns. Column stats are simple statistics that describe the size of the compressed column internals and are saved to the tenant directory per column.

Enhancements also include better row filtering to improve performance when reading the following:

Columns from parquet segments that contain multiple duplicate rows
Columns from tables with primary keys
Columns that are evicted before reading

Note

These enhancements are available in a new beta version of the feature.

This feature is disabled by default. To enable it, contact Incorta Support to edit the engine.properties file in both the Analytics Service and Loader Service nodes and enable the required options.

This feature requires some conditions to be true, for example, the number of rows must be greater than or equal to 10,000,000. The row count affects the number of threads that Incorta uses to read the column data.

New schema public API endpoints

Two new public API endpoints are available to load physical schemas and retrieve the details of a physical or business schema.

/schema/load: This endpoint starts the load of one or more physical schemas. For more information, refer to Public API → Load Schema Endpoint.
/schema/{schemaName}/status: This endpoint retrieves the details of a specific physical or business schema. Details include the available objects (tables or views), the number of columns per object, creation and modification time, and schema owner. For physical schemas, it returns the status of the most recent model update and load jobs and the data size. For more information, refer to Public API → Schema Status Endpoint.

Support for Spark 3.3

Incorta Cloud now supports Spark 3.3. Newly created clusters will use the new Spark version. However, existing clusters that you upgrade to the 2022.11.0 release will continue to use the existing Spark version until further notice.

Spark 3.3 provides the following benefits: