Release Notes 2022.11.x
Release Highlights
Incorta introduces several improvements to data ingestion and data manipulation in this release. These improvements include a next-generation loader with a new architecture, a new connector for the Oracle Warehouse Management System (WMS), and support for fiscal calendars.
This release includes preview features, such as scheduled load plans with multiple schemas, scheduled dashboard delivery with bursting reports based on the recipient’s context, and extending parallel reading of columns into memory for all data types.
Additional features include support for Spark 3.3 on newly created clusters, enhancements to the Bubble visualization, and two new public API endpoints to load schemas and retrieve their details.
The 2022.11.1 and 2022.11.2 maintenance packs include additional enhancements and fixes.
The 2022.11.0 release is available for newly created clusters and new trial users. If you’re an existing customer upgrading from a prior release, please contact Incorta Support to schedule your upgrade.
This release uses the Data Agent version 5.2.0. Please make sure to upgrade to the data agent version available on the Cloud Admin Portal.
Upgrade considerations
daysBetween function
In this release, the daysBetween function does not automatically round up the returned value to the nearest long value, instead, it returns and saves the value as double. To round the returned value and retain the old behavior, you have to wrap up the daysBetween function with the round or ceil function. To avoid errors that might occur after the upgrade when rendering insights and business schemas that reference columns with the daysBetween function, you have to load from staging all physical schemas where these columns exist.
Schemas with user-defined load order
With the introduction of the new-generation loader, Incorta automatically detects inter-object dependencies within a load plan during the Planning phase of a load job. The Loader Service utilizes these detected dependencies and the user-defined load order within the schema to create an execution plan for loading objects. However, it’s important to note that using both the MVs' user-defined load order and automatically detected dependencies may result in an execution plan with cyclic dependencies, leading to load job failures. To avoid such failures, it is recommended to delete the MVs' user-defined load order before upgrading to the 2022.11.0 release.
New Features
Architecture and Application Layer
Data Management Layer
Dashboards, Visualizations, and Analytics
Integrations
Refer to What's New in Incorta Cloud 2022.11.0 for more details and examples.
Next-generation loader
Incorta's next-generation loader comes with a new architecture that comprises multiple components. The new architecture effectuates the plan-based execution of load jobs by introducing a new Planning phase. During this phase, the Loader Service detects the dependencies among the objects in the load job, along with manually defined load groups, and creates a plan for loading objects accordingly. In addition, the Loader Service concurrently executes the extraction and transformation of objects in the load job in the same phase: the Running phase. The new architecture optimizes the Post-load calculations so that the Loader Service concurrently executes different calculation types. These enhancements eliminate previous limitations and open the door for more improvements.
Parallel column reading enhancements
Incorta now extends the parallel reading of columns into memory to include numeric and date data types. In addition, Incorta can store and use column stats of parquet files to optimize reading and loading compressed columns. Enhancements also include better row filtering to improve performance when reading columns from parquet segments that contain multiple duplicate rows, columns from tables with primary keys, or columns that are evicted before reading.
The enhancements are available in a new beta version of the feature, which is disabled by default. Contact Incorta Support to enable it.
Oracle WMS Cloud connector
The new Oracle Warehouse Management Software (Oracle WMS Cloud) application connector brings your logistics data into Incorta so you can start investigating your data and building insights. Oracle WMS Cloud is the logistics system in the Oracle Supply Chain Management and Manufacturing system.
Multi-schema load plans
As a Schema Manager, you can now schedule load plans that involve loading multiple physical schemas. The benefits of this feature include the following:
- Reducing the number of required load plans
- Expediting and facilitating data refresh cycles
- Solving dependency issues among schemas
- Minimizing the total loading time as the Loader Service calculates cross-schema joins for all schemas in the load plan only once
Multi-schema loading is a preview feature in this release. To enable it on your cluster, contact Incorta Support.
New schema public API endpoints
Two new public API endpoints are available to load physical schemas and retrieve the details of a physical or business schema.
- /schema/load: This endpoint starts the load of one or more physical schemas. For more information, refer to Public API → Load Schema Endpoint.
- /schema/{schemaName}/status: This endpoint retrieves the details of a specific physical or business schema. Details include the available objects (tables or views), the number of columns per object, creation and modification time, and schema owner. For physical schemas, it returns the status of the most recent model update and load jobs and the data size. For more information, refer to Public API → Schema Status Endpoint.
Support for fiscal calendars
To empower data engineers in the financial sector, Incorta now natively supports fiscal year calendars. This feature simplifies the setup and consumption of fiscal calendars in Incorta and allows for the rollup of transactional data into fiscal calendars. You can define the start of the fiscal year in the Cluster Management Console (CMC) under Server Configurations > Customization.
Date filters and date aggregations now honor fiscal calendars. In addition, multiple fiscal-calendar-based system variables are now available to select from when creating filters, formula columns, or queries. You can also apply different fiscal-calendar formats depending on the selected Date Part. You can use these new variables and date parts along with the existing ones to create comparisons between fiscal and Gregorian calendars without much effort. The ago and toDate functions now support the fiscal calendar as well.
Bursting reports
Incorta now supports security context-based bursting reports, where reports are distributed based on the recipients’ context instead of the sender’s context. For example, if salespeople in different regions each need a report showing the sales target for their country or region, use burst reports to send each salesperson only the information they need.
To burst a report, in the dashboard Send/Schedule a Report page, select the Burst Report check box. You can send or schedule a bursting report or dashboard for up to 300 internal users with Incorta accounts and at least view access rights to the dashboard. The report runs once but renders customized data based on the security filters or session variables of each recipient.
Bursting reports is a preview feature in this release. To enable this feature on your cluster, contact Incorta Support.
Bubble chart size enhancements
You can now set minimum and maximum percentage values for the radii of the bubbles within a bubble visualization. You can do that using the newly added property Radius in the pill properties of Size By tray.
Support for Spark 3.3
Incorta Cloud now supports Spark 3.3. Newly created clusters will use the new Spark version. However, existing clusters that you upgrade to the 2022.11.0 release will continue to use the existing Spark version until further notice.
Spark 3.3 provides the following benefits:
- Improved join performance via Bloom filters
- Pandas API support
- ANSI compliance enhancements for Spark SQL
- Error message improvements
For more details on Spark 3.3, refer to Spark documentation.
Additional enhancements and fixes
In addition to the new features and major enhancements mentioned above, this release introduces some additional enhancements and fixes that help to make Incorta more stable, engaging, and reliable.
This table shows the additional enhancements.
| Enhancement | Area |
|---|---|
The DaysBetween() function now returns a double value enabling you to use the round or ceil functions to return an integer. | Built-in Functions |
| New clusters will now have insight over insights enabled by default. | Cluster Management Console |
A new (𝒙) button is now available in the filters screen to facilitate retrieving available variables instead of entering the $ dollar sign manually. | Dashboards |
| An error message will now display when the user attempts to create a scheduled load plan without selecting the desired days of the week. | Physical Schemas |
| A new Headless Rendering Timeout option is now available in the Server Configurations to avoid timeout errors while sending scheduled dashboards. | Scheduler |
This table shows the issues that this release fixes.
| Issue resolved | Area |
|---|---|
$currentWeekStart and $lastWeekStart system variables returned incorrect day and year for the first week of the year. | Analyzer |
| The formula of a parent joined table did not appear in the formula builder when attempting to create a nested formula in the child table. | Analyzer |
| Changing the regional format from the CMC did not update the user interface. | Cluster Management Console |
| With chunking enabled, CSV file rows could be truncated when using languages that used multibyte characters. | Connectors |
| The Fusion connector would fail to load successfully with an UnknownHostException error. | Connectors |
| SQL connectors did not execute pre-SQL statements. | Connectors |
| Oracle Fusion connector dropped records during extraction. | Connectors |
| Schema load failed when discovering empty files while using Oracle Fusion. | Connectors |
| Incremental load in Oracle Fusion retrieved zero records. | Connectors |
| Shared dashboard folder appeared empty when favorited twice in quick succession. | Dashboards |
| Imported visualizations from prior Incorta versions that included negative values did not correctly display data. | Dashboards |
| A Super User couldn’t import a dashboard with bookmarks and overwrite an existing one despite having edit access rights to it. | Dashboards |
| After applying changes to filter options in a dual KPI, the changes were not automatically updated until the user navigated away from the insight. | Dashboards |
| Some insight names displayed HTML tags on the data lineage page. | Inspector Tool |
| In an Incorta Analyzer table, formulas in a distinct filter caused schema updates to fail. | Physical Schemas |
| Schema loads where a materialized view was dependent on a very large table could fail to load. | Physical Schemas |
| A schema load when using a data source vs local files would produce an inconsistent load success. | Physical Schemas |
| While using Incorta users would intermittently be logged out. | Security |
| Table visualizations with one or more empty column headers would not be sorted correctly. | Visualizations |
| Missing rows from hierarchy tables | Visualizations |
| After duplicating a slicer insight, sort and aggregate filters would be removed from the duplicate. | Visualizations |
| Some descriptions in the formula builder were no longer displayed after an upgrade. | Visualizations |
2022.11.1 Maintenance Pack
Enhancements
| Enhancement | Area |
|---|---|
| Data Lineage is now a General Availability (GA) feature that is ready for production use. | Data Lineage |
Executing Spark jobs in cluster deploy mode is now more stable and reliable. | Materialized Views |
| Some data applications now contain third-party sample dashboards for specific visualization tools, such as Excel, PowerBI, and Tableau. You can use the Visualization Tool new filter option in the Marketplace to filter out the data application list according to the visualization tool. | Marketplace |
| Multiple improvements to the MV Assistant are now available. ● Enhanced how the tool filters out files to analyze ● A more accurate number of executors based on the actual application resources assigned in runtime ● Fixed some issues that might cause the MV Assistant to fail to calculate the recommended values for the Spark-related configurations | MV Assistant |
Fixed Issues
| Issue resolved | Area |
|---|---|
| Users couldn’t access a 2022.11.0-cluster using Safari. | Cloud Platform |
Testing the connection to an Oracle Cloud Application (Fusion) data source threw an error: INC_01020102: System Error. Please ask Administrator to check the log files. | Connectors |
| Dashboards of trial data applications showed no data. | Data Applications |
Materialized views based on tables with some date functions, such as addDays, failed to create. | Materialized Views |
| Incorta Analyzer Table built on an aggregated table showed incorrect values. | Physical Schemas |
Updates on existing schemas failed to save with an error: java.lang.IndexOutOfBoundsException: Index: -1 | Physical Schemas |
| When multi-schema loading was enabled and one of the schemas failed in the Post-load stage, all schemas showed the Finished with Error status even if other schemas loaded successfully. | Physical Schemas |
| An incremental load job on a schema skipped PostgreSQL MVs that referenced business views and used a parameterized query statement. In addition, the MV load type showed full load (F) instead of incremental (I) | Physical Schemas |
SQLi queries did not release their locks when the Order by was based on a formula, which caused dashboard renders to throw an error: INC_04050702: the underlying data for this insight is being updated, Cannot read lock on Resources... | SQLi |
2022.11.2 Maintenance Pack
| Issue resolved | Area |
|---|---|
Intermittent tables based on Google Sheets failed with an error: No SpreadSheet with name… found. | Connectors |
| Load from staging didn’t force all columns to be evicted from memory, making it hard to recover from inconsistency or corruption issues without restarting the server. | Physical Schemas |
| Error handling for column loading was missing interruption handling, causing some columns to fail and some others to get stuck in an undefined and inconsistent state till the Loader Service was restarted. | Physical Schemas |
| Minor issues with logging and monitoring | Server |
Known issues
The following are the known issues in 2022.11.0.
- An incremental load job on a schema skips PostgreSQL MVs that reference business views and use a parameterized query statement. In addition, the MV load type shows full load (F) instead of incremental (I). (Resolved in 2022.11.1)
- When multi-schema loading is enabled, adding a load plan from Schema Manager doesn’t display the total number of schemas correctly.
2022.11.1 Known issues
- The Visualization Tool filter section doesn't appear correctly when changing the user interface language.
For information about all the known issues and workarounds in Incorta Cloud releases, refer to Known Issues in 2022 Releases.