Guides → Configure Server

You can manage your server through the Cluster Management Console (CMC) by configuring various settings, such as an SQL interface port, connection pool size, proxy read timeout, and other settings. Here are the steps to configure your server:

  • Sign in to the CMC.
  • In the Navigation bar, select Clusters > Cluster-name.
  • In the Canvas tabs, select Cluster Configurations > Server Configurations.

The following server configurations are available in the CMC:

Important

To avoid losing data after configuring your settings on a page, select Save before navigating to another page.

Clustering

The following table illustrates the cluster configuration property:

Configuration PropertyControlRequires RestartDescription
Kafka Consumer Service Nametext boxYes, Loader service onlyEnter the name of the Loader service that acts as a Kafka consumer in the following format: <NODE_NAME>.<SERVICE_NAME>

Note the following:
  ●   If you do not configure the name of the Kafka Loader service, a random loader service will be automatically assigned leading to unexpected results.
  ●   If you change the Kafka Loader service consumer from one Loader service (A) to another Loader service (B), you must restart the current Loader service (A) first, and then restart the Loader service (B).

SQL Interface

Incorta can act as a PostgreSQL database using the Structured Query Language Interface (SQLi). Thus, you can utilize Incorta’s powerful engine performance and features through other BI tools, such as Tableau.

The following table illustrates the configuration properties of the SQL interface:

Configuration PropertyControlRequires RestartDescription
SQL interface portspin boxYes, Analytics service onlyEnter a port number to connect to the Incorta Engine. The Engine mode uses this port and runs the SQL interface queries against the data loaded in memory.
Data Store (DS) portspin boxYes, Analytics service onlyEnter a port number to run queries directly using Spark. The Spark mode (Parquet mode) runs the SQL interface queries against the data loaded in the staging area using Incorta compacted parquet files.
New Metadata handler portspin boxYes, Analytics service onlyEnter a port number for an auxiliary process to resolve metadata for SQLI clients.
Enable SSL for SQL interface portstoggleNoEnable this property to use Secure Sockets Layer (SSL) for SQLI connections. You must provide the SSL certificate details (path and passphrase).
SSL certificate(JKS) Path used for SQL interfacetext boxNoEnter the path to SQLI SSL certificate.

This property is only available when Enable SSL for SQL interface ports is toggled on.
SSL certificate(JKS) passphrase used for SQL interfacetext boxNoEnter the passphrase to SQLI SSL certificate.

This property is only available when Enable SSL for SQL interface ports is toggled on.
Enforce Accepting Only SSL ConnectionstoggleNoEnable this property to allow SSL connections only.

This property is only available when Enable SSL for SQL interface ports is toggled on.
Connection pool sizespin boxYes, Analytics service onlyEnter the number of pool connections. This creates multiple connections that improve the query response time.
External Tables CSV file pathtext boxYes, Analytics service onlyEnter the CSV file path of your external tables.
Enable CachetoggleNoEnable this property to cache recurring SQL operations and enhance the performance of query execution, if there is an available cache size.
Cache size (In gigabytes)spin boxNoEnter the maximum caching size (in GBs) of the data returned by the SQLi queries. When the cache size exceeds the limit, the least recently used (LRU) data is evicted to avail space for newer cache.

You must set the cache size based on the available memory on the Incorta host server, and the size of the common query results.
Cached query result max sizespin boxNoEnter the maximum threshold size for each query result, which is the table cell count (rows multiplied by columns).

If the size of a query result exceeds the threshold value, it will not be cached.
Enable cache auto refreshtoggleNoEnable this property to automatically refresh the cache at specified intervals.
Refresh cache interval (In minutes)spin boxNoEnter the interval in minutes to automatically refresh the cache.

This property is only available when Enable cache auto refresh is toggled on.
Use column labels instead of column namestoggleNoEnable this property to use table columns labels instead of column names during the SQL execution. This affects the column list specified in the query and the returned column names in the result.
SQL Engine CachetoggleNoEnable this property to use caching for SQL engine result set to speed up reptitive operations performed during the processing and the execution of a query. The default cache size is 5% of the off-heap memory.

Spark Integration

Incorta utilizes Spark to execute complex queries that are not currently supported and to perform queries on data residing in the staging area without loading those data into Incorta’s memory.

The following table illustrates the Spark integration configuration properties:

Configuration PropertyControlRequires RestartDescription
Spark master URLtext boxYes, all servicesEnter the Spark Master URL for the Apache Spark instance to execute materialized views (MVs) or SQL queries.

Here are the steps to acquire the Spark Master information:
  ●   Navigate to the Spark host server UI (from any browser), using the following format: <SPARK_HOST_SERVER>:<SPARK_PORT_NO>
  ●   Copy the Spark Master connection string in the format: spark://<CONNECTION_STRING>:<SPARK_PORT_NO>
  ●   The default port number for installed Spark on Incorta is 7077.
Enable a custom Delta Lake ReadertoggleNoEnable this property to instruct Spark to use the Incorta Custom Delta Lake Reader to read data from the Delta Lake files instead of using Spark native Delta Lake reader . The Incorta Custom Delta Lake Reader is optimized for queries on relatively small data sets.
Switching between these two readers may cause the first SQLi query on the Spark port to consume more time because the SQLi engine will be restarted.
Enable SQL ApptoggleNoEnable this property to start the SQL App, and then use Engine and Parquet modes to execute incoming SQL queries.
Enable spark fallbacktoggleYes, Analytics serviceThis property is enabled by default. Disabling this feature prevents the In-Memory engine from falling back to the Spark or Parquet mode when the query is not executed, instead, an SQL error is returned.

This property is only available when Enable SQL App is toggled on.
SQL App driver memoryspin boxNoEnter the app drive memory (in GBs) to be used by the SQL interface Spark. This memory is only used for constructing (not calculating) the final results. Consult with the Spark admin to set this value.
SQL App coresspin boxNoEnter the number of dedicated Central Processing Unit (CPU) cores reserved for the SQL interface Spark application.

Ensure that there are enough cores for the OS, other services, and applications.
SQL App memoryspin boxNoEnter the maximum memory (in GBs) used by SQL interface Spark queries.

Note the following:
  ●   You must set an adequate memory size when using materialized views.
  ●   The memory required for both applications combined cannot exceed the Worker Memory.
SQL App executorsspin boxNoEnter the maximum number of executors that can be spawned on a single worker.

Each of the executors will allocate some of the cores defined in sql.spark.application.cores, and will use part of the memory defined in sql.spark.application.memory .

The cores and memory assigned per executor are equal. Thus, the number of SQL App cores and SQL App memory is divided by the number of executors. If the number of executors is greater than the SQL App memory, the executors will consume the assigned memory at 1 GB/executor.

Examples:
  ●   If the SQL App cores = 7 and the SQL App executors = 3, each executor will take 2 cores, and 1 of the cores will not be utilized.
  ●   If the SQL App memory = 5 and the SQL App executors = 7, then 5 cores will be created, with 1 GB each.
SQL App shuffle partitionsspin boxNoEnter the number of SQL interface shuffle partitions. A single shuffle represents a block of data being processed to perform joins and/or aggregations.

Note the following:
  ●   The size of a shuffle partition increases as the size of processed data increases.
  ●   The optimum shuffle partition size is around 128 MBs.
  ●   Increase the shuffle partitions as the processed data size increases. However, an increase in shuffle partitions can lead to an increase in CPU utilization.
  ●   If a query operates on a trivial amount of data, an increased amount of partitions will lead to a small partition size which may increase the query execution time due to the overhead of managing the unnecessary amount of partitions. Inadequate partition size may cause the query to fail.
SQL App extra optionstext boxNoEnter the extra Spark options that you want to pass to the SQL interface Spark bridge application. These options can be used to override default configurations.

Here are sample values: spark.sql.shuffle.partitions=8; spark.executor.memory=4g; spark.driver.memory=4g
Enable SQL App Dynamic AllocationtoggleYes, Analytics serviceEnable this property to dynamically allocate the Data Hub Spark application by adjusting the number of application executors based on the workload.

This property optimizes resource utilization as it removes idle executors, and claims them again if the workload increases.
Spark App portspin boxYes, Analytics serviceEnter the port number to connect Incorta to Spark and access the Data Hub.
Spark App control channel portspin boxNoEnter the control channel port number to send a shutdown signal to the Spark SQL app when the Incorta server requires Spark to shut down.
SQL App spark home (Optional)text boxNoOptionally, enter the file system path for the Apache Spark instance that executes queries sent to the Spark App (Engine and Parquet modes).

Note the following:
  ●   The file system path is required for linking necessary libraries shipped by Apache Spark.
  ●   Without a file system path, the "SPARK_HOME" environment variable is looked up instead.
  ●   If the "SPARK_HOME" variable is not set, the value of the Spark instance for Incorta MVs and compaction is used instead.
  ●   Else, an exception is thrown when the file system path is required to serve an SQL interface Spark (Engine or Parquet mode) query.
SQL App Spark master URL (Optional)text boxNoOptionally, enter the URL of the Spark master server that executes the SQL interface queries over Incorta Engine or Parquet.

The default URL is the Incorta-shipped Spark master URL.
Materialized view application coresspin boxNoEnter the number of CPU cores for MVs. The default value is 1.

The allocated cores for all running Spark applications cannot exceed the dedicated cores for the cluster unless Dynamic Allocation is enabled. Thus, the value will be used to compute the CPU cores for the initial executors.
Materialized view application memoryspin boxNoEnter the maximum memory size (in GBs) to use for MVs. The default is 1 GB.

The memory for all Spark applications combined cannot exceed the cluster memory.
Materialized view application executorsspin boxNoEnter the maximum number of executors that can be spawned by a single materialized view application.

Each of the executors will allocate some of the cores defined in sql.spark.mv.cores, and will consume part of the memory defined in sql.spark.mv.memory.

The cores and memory assigned per executor are equal. For example, configuring an application with cores=4, memory=8, executors=2, will result in spawning 2 executors, in which each executor consumes 2cores/4GB from the cluster.
Materialized view driver memoryspin boxNoEnter the memory size (in GBs) to use in the MV driver process. The default value is 1 GB.
Enable dynamic allocation in MVstoggleNoEnable this property to dynamically allocate resources in MVs by scaling the number of application executors up and down based on the workload.

This property requires running the Spark shuffle service.
Enable automatic ML library downloadtoggleNoEnable this property to automatically download the Machine Learning (ML) library or use the bundled one.

Tuning

The following table illustrates the tuning configuration properties:

Configuration PropertyControlRequires RestartDescription
Enable parquet file CompressiontoggleNoEnable this property to compress your parquet files.

It is necessary to compact parquet files when using Materialized views.
Proxy read timeoutspin boxYes, Analytics service onlyEnter the required read timeout for rendering a UI request in seconds. You can adjust the value based on the proxy server configurations.
Cleanup Job Refresh Time in Minutesspin boxYes, Loader service onlyEnter the time interval in minutes for a cleanup job schedule to remove unnecessary files from the shared storage.

The interval value cannot be less than 10 minutes.

UI Customizations

The following table illustrates the configuration properties of the User Interface (UI) customizations:

Configuration PropertyControlRequires RestartDescription
Color Palette Modedrop down listNoChoose a color palette for your insights and dashboards. The options are:
  ●   Sophisticated
  ●   Classic
  ●   Contemporary
  ●   Bright & Bold
  ●   90s Retro
  ●   Custom
Custom Palette Colortext boxNoEnter the HEX codes for 12 colors to be used by the custom palette as follows: #255888, #D1481A, #285F6C,...

This property is only available when you set Color Palette Mode to Custom.

Diagnostics

The following table illustrates the diagnostics configuration property:

Configuration PropertyControlRequires RestartDescription
Loggingtext boxNoEnter the logging level to specify the required information in your log files.

Here are the available logging levels in descending order:
  ●   OFF - disables logging
  ●   SEVERE (highest value) - this level logs severe events that can cause termination.
  ●   WARNING
  ●   INFO
  ●   CONFIG
  ●   FINE
  ●   FINER
  ●   FINEST (lowest value)
  ●   ALL - logs all messages

The expression must be as follows: component1=level1:component2=level2.

For example, you can reduce the level of logging on your production system for the Loader service by setting the logging level to SEVERE. Enter: com.incorta.engine=SEVERE:com.incorta.server=SEVERE:com.incorta.loader=SEVERE
Allow Cloud Disk Space MonitoringtoggleNoDisabled by default. This option applies to cloud storage only, enabling it would monitor the disk space every 5 min which is not recommended as it might increase number of I/O operations.

Data Agent

The following table illustrates the data agent configuration property:

Configuration PropertyControlRequires RestartDescription
Enable Data AgenttoggleYes, all servicesEnable this property to allow Incorta to connect and extract data from one or more data sources behind a firewall.

For more information, refer to the Concepts → Data Agent document.
Analytics Data Agent Portspin boxYes, Analytics service onlyEnter the port number of your Analytics data agent. The value cannot be less than 1 or greater than 65535. This property is only available when Enable Data Agent is toggled on.
Loader Data Agent Portspin boxYes, Loader service onlyEnter the port number of your Loader data agent. The value cannot be less than 1 or greater than 65535.

This property is only available when Enable Data Agent is toggled on.
Analytics Public Hosts and Portstext boxNoEnter your Analytics public hosts and ports.

Note the following:
  ●   Changing the Analytics public hosts and ports requires re-generating the .auth files
  ●   This property is only available when Enable Data Agent is toggled on.
Loader Public Hosts and Portstext boxNoEnter your Loader public hosts and ports.

Note the following:
  ●   Changing the Loader public hosts and ports requires re-generating the .auth files
  ●   This property is only available when Enable Data Agent is toggled on.

Email

The following table illustrates the email configuration properties:

Configuration PropertyControlRequires RestartDescription
Local Rendering URL Protocoldrop down listNoSelect the local rendering URL protocol:
  ●   http
  ●   https
Local Rendering Hosttext boxNoEnter the local rendering host. For example, 127.0.0.1.
Local Rendering Portspin boxNoEnter the local rendering port number. For example, 8080.
Headless Browser Rendering (in milliseconds)spin boxNoEnter the number of milliseconds after which rendering a dashboard would timeout. For example, 9000.

Security

The following table illustrates the security configuration properties:

Configuration PropertyControlRequires RestartDescription
Enable Iframe inclusiontoggleNoEnable external websites to host Incorta dashboard(s) in an iFrame. Enabling this option mandates that Incorta is accessed through SSL connections (HTTPS).