References → Scalability Modes
The Spark scalability modes control how the Advanced SQLi accesses optimized tables, non-optimized tables, and verified business views.
The available modes are:
ddm-preferred
(default)parquet-preferred
ddm-only
The following table shows the source that the Advanced SQL service uses to read the object data for each mode:
Object/Mode | ddm-preferred | parquet-preferred | ddm-only |
---|---|---|---|
Optimized tables with security filters, formula columns, or encrypted columns | DDM | DDM | DDM |
Optimized tables without security filters, formula columns, or encrypted columns | DDM | Parquet | DDM |
Partially optimized tables with security filters, formula columns, or encrypted columns | DDM | DDM | DDM |
Partially optimized tables without security filters, formula columns, or encrypted columns | DDM | Parquet | DDM |
Non-optimized tables with security filters, formula columns, or encrypted columns | Not supported as of 2024.1.3 | Not supported as of 2024.1.3 | Not supported as of 2024.1.3 |
Non-optimized tables without security filters, formula columns, or encrypted columns | Parquet (as non-optimized tables do not have DDM files) | Parquet | Not supported (Querying these tables returns no data.) |
Verified Business Views referencing optimized tables | DDM (of the source table) | DDM | DDM |
Verified Business Views referencing non-optimized tables | Not supported as of 2024.1.3 | Not supported as of 2024.1.3 | Not supported as of 2024.1.3 |
- As the DDM files of partially optimized tables may not have all the data existing on the Parquet files, the results of querying a partially optimized table may vary depending on the selected scalability mode.
- Business Views based on non-optimized tables are not supported. While these views will be discoverable via the Advanced SQLi, querying them will return no data or throw an error according to the scalability mode used.
You can use the EXPLAIN
statement to check if the query uses the DDM or Parquet files.
Example: EXPLAIN SELECT * from schemaName.tableName
- If the response includes
BatchScan,
the query uses the DDM files. - If the response includes
FileScan parquet
, the query uses the Parquet files.
Set the scalability mode
The default scalability mode is ddm-preferred
. However, you can set the scalability mode individually per query or globally per service. The query setting overrides the service settings.
Set the mode per query
If the external tool supports using the SET
command, add the following before the query to set the scalability mode:
SET spark.incorta.scalability.mode=[mode]
Example: SET spark.incorta.scalability.mode=parquet-preferred
Set the mode per service
To set or change the scalability mode per service:
- Navigate to the following path:
/<incorta_installation_path>/IncortaNode/kyuubi/services/<service_GUID>/conf/
. - Edit the
kyuubi-defaults.conf
to add or edit the following property and set the mode as required:spark.incorta.scalability.mode=[mode]
. - Restart the Advanced SQL service.