1. Home
  2. Previous Releases
  3. Latest Release
  4. Guides
  5. Concepts
  6. Tools
  7. References
    1. Apache Spark
    2. Built-in Functions
    3. Incorta Component SDK
    4. Incorta Connector SDK
    5. Connectors
    6. Public API
    7. Security
    8. SQL Interface
      1. SQL Interface
      2. Configure Spark with SQLi
      3. Advanced SQL Interface
        1. Setup and Configuration
        2. SparkX Metastore Sync
        3. Discovery of Non-Optimized Tables
        4. SparkX Access to Cloud Storage
        5. Data Extraction via Advanced SQLi and Public API
        6. Scalability Modes
        7. Advanced SQLi Ports
        8. Custom SparkX Sizes
    9. Visualizations
    10. Other
  8. Integrations
  9. Data applications
  10. Releases

References → Scalability Modes

The Spark scalability modes control how the Advanced SQLi accesses optimized tables, non-optimized tables, and verified business views.

The available modes are:

  • ddm-preferred (default)
  • parquet-preferred
  • ddm-only

The following table shows the source that the Advanced SQL service uses to read the object data for each mode:

Object/Modeddm-preferredparquet-preferredddm-only
Optimized tables with security filters, formula columns, or encrypted columnsDDMDDMDDM
Optimized tables without security filters, formula columns, or encrypted columnsDDMParquetDDM
Partially optimized tables with security filters, formula columns, or encrypted columnsDDMDDMDDM
Partially optimized tables without security filters, formula columns, or encrypted columnsDDMParquetDDM
Non-optimized tables with security filters, formula columns, or encrypted columnsNot supported as of 2024.1.3Not supported as of 2024.1.3Not supported as of 2024.1.3
Non-optimized tables without security filters, formula columns, or encrypted columnsParquet (as non-optimized tables do not have DDM files)ParquetNot supported (Querying these tables returns no data.)
Verified Business Views referencing optimized tablesDDM (of the source table)DDMDDM
Verified Business Views referencing non-optimized tablesNot supported as of 2024.1.3Not supported as of 2024.1.3Not supported as of 2024.1.3
Notes
  • As the DDM files of partially optimized tables may not have all the data existing on the Parquet files, the results of querying a partially optimized table may vary depending on the selected scalability mode.
  • Business Views based on non-optimized tables are not supported. While these views will be discoverable via the Advanced SQLi, querying them will return no data or throw an error according to the scalability mode used.
Recommendation

You can use the EXPLAIN statement to check if the query uses the DDM or Parquet files.

Example: EXPLAIN SELECT * from schemaName.tableName

  • If the response includes BatchScan, the query uses the DDM files.
  • If the response includes FileScan parquet, the query uses the Parquet files.

Set the scalability mode

The default scalability mode is ddm-preferred. However, you can set the scalability mode individually per query or globally per service. The query setting overrides the service settings.

Set the mode per query

If the external tool supports using the SET command, add the following before the query to set the scalability mode:

SET spark.incorta.scalability.mode=[mode]

Example: SET spark.incorta.scalability.mode=parquet-preferred

Set the mode per service

To set or change the scalability mode per service:

  • Navigate to the following path: /<incorta_installation_path>/IncortaNode/kyuubi/services/<service_GUID>/conf/.
  • Edit the kyuubi-defaults.conf to add or edit the following property and set the mode as required: spark.incorta.scalability.mode=[mode].
  • Restart the Advanced SQL service.