1. Home
  2. Previous Releases
  3. Latest Release
  4. Guides
  5. Concepts
  6. Tools
  7. References
    1. Apache Spark
    2. Built-in Functions
    3. Incorta Component SDK
    4. Incorta Connector SDK
    5. Connectors
    6. Data Studio
    7. Public API
    8. Security
    9. SQL Interface
      1. SQL Interface
      2. Configure Spark with SQLi
      3. Advanced SQL Interface
        1. Setup and Configuration
        2. SparkX Metastore Sync
        3. Discovery of Non-Optimized Tables
        4. SparkX Access to Cloud Storage
        5. Data Extraction via Advanced SQLi and Public API
        6. Scalability Modes
        7. Advanced SQLi Ports
        8. Custom SparkX Sizes
        9. Notebook for Business Users
    10. Visualizations
    11. Other
  8. Integrations
  9. Data applications
  10. Releases
  11. Incorta Premium

References → Custom SparkX Sizes

For on-prem installations, there are minimum or recommended sizes (for both memory and CPU) to use with custom Spark depending on your cluster size.

The following table illustrates the recommended sizes to use:

Cluster size/ ServiceXSmallSmallMediumLargeXLarge2XLarge
Kyuubi Server PodDefault for both CPU and MemoryDefault for both CPU and MemoryDefault for both CPU and MemoryDefault for both CPU and MemoryDefault for both CPU and MemoryDefault for both CPU and Memory
SparkX Driver Pod
  ●   Memory: 16
  ●  CPU Cores: 4
  ●  MaxResultSize: 200 MBs
  ●   Memory: 16
  ●  CPU Cores: 4
  ●  MaxResultSize: 200 MBs
  ●   Memory: 32
  ●  CPU Cores: 8
  ●  MaxResultSize: 400 MBs
  ●   Memory: 64
  ●  CPU Cores: 16
  ●  MaxResultSize: 800 MBs
  ●   Memory: 64
  ●  CPU Cores: 16
  ●  MaxResultSize: 800 MBs
  ●   Memory: 64
  ●  CPU Cores: 16
  ●  MaxResultSize: 800 MBs
SparkX Executors
  ●   Memory: 8
  ●  CPU Cores: 2
  ●  Min Executors: 0
  ●  Max Executors: 1
  ●   Memory: 8
  ●  CPU Cores: 2
  ●  Min Executors: 0
  ●  Max Executors: 4
  ●   Memory: 32
  ●  CPU Cores: 8
  ●  Min Executors: 0
  ●  Max Executors: 6
  ●   Memory: 32
  ●  CPU Cores: 8
  ●  Min Executors: 0
  ●  Max Executors: 8		Memory: 32
  ●   CPU Cores: 8
  ●  Min Executors: 0
  ●  Max Executors: 8
  ●   Memory: 32
  ●  CPU Cores: 8
  ●  Min Executors: 0
  ●  Max Executors: 8
Engine Partitioning/Data Transfer Parallelism81632646464