1. Latest Release Home
  2. Guides
  3. Concepts
    1. Aggregate filter
    2. AI App
    3. AI Containers
    4. Alias
    5. Apache Avro™
    6. Apache Kafka™
    7. Applied filter
    8. Base table
    9. Bookmark
    10. Business schema
    11. Business schema view
    12. Business schema view column
    13. Business schema view formula column
    14. Column
    15. Connector
    16. Dashboard filter
    17. Dashboard folder
    18. Dashboard runtime filter
    19. Data agent
    20. Data alert
    21. Data destination
    22. Data purge
    23. Data retention
    24. Date system variable
    25. Derived variable
    26. Direct query
    27. Distinct filter
    28. Dynamic field
    29. Expression
    30. External data source
    31. External session variable
    32. External table
    33. Filter expression
    34. Filter expression session variable
    35. Filter option
    36. Formula column
    37. Formula expression
    38. Full load
    39. Global variable
    40. Incorta Analyzer Table
    41. Incorta Analyzer Table column
    42. Incorta Analyzer Table formula column
    43. Incorta Analyzer View
    44. Incorta Analyzer View formula column
    45. Incorta RAG
    46. Incorta Spark SQL View
    47. Incorta SQL Table
    48. Incremental Load
    49. Individual filter
    50. Insight filter
    51. Internal query expression
    52. Internal session variable
    53. Join
    54. Join condition
    55. Join condition column
    56. Join filter
    57. Join filter column
    58. Level-based measure
    59. Load filter
    60. Local data file
    61. Local data folder
    62. Log-based incremental load
    63. Materialized View
    64. Materialized View column
    65. Materialized View formula column
    66. Measure filter
    67. Miscellaneous system variable
    68. Multi-source table
    69. On-demand loader
    70. Physical schema
    71. Physical schema table
    72. Physical schema table column
    73. Physical schema table formula column
    74. Physical schema table runtime security filter
    75. PostgreSQL (Incorta SQL) View
    76. Presentation variable
    77. Prompt
    78. Query system variable
    79. Remote table
    80. Runtime business view
    81. Runtime business view formula column
    82. Runtime security filter
    83. Schema evolution
    84. Service Status Observability
    85. Session variable
    86. Shared Storage
    87. Snowflake result set
    88. Snowflake view
    89. Upgrade Readiness
    90. Variable
    91. Visualization
  4. Tools
  5. References
  6. Integrations
  7. Data applications
  8. Data Governance
  9. Release Notes
  10. Incorta Premium
  11. Release Support Policy
  12. Release Model

Concepts → On-Demand Loader

About

In deployments with large schemas or heavy ETL workloads, a single always-on loader is often over-provisioned for most of the day. High-specification loader nodes (memory and CPU) run continuously while large load plans execute only a few times daily, resulting in unnecessary infrastructure costs.

The on-demand loader is a Loader Service node that starts automatically when a load plan requires it and shuts down automatically after a configurable idle period. Unlike the always-on primary loader, an on-demand loader is not running continuously; it is activated only for the load plans assigned to it, reducing infrastructure costs for deployments where high-capacity loading is needed infrequently.

Feature availability

This feature is available on Incorta Cloud deployments only.

On-demand loader architecture

The on-demand loader introduces a dual-loader architecture:

  • Primary loader: Always running. Handles frequent, lightweight workloads such as incremental loads and schema updates. This loader is required; at least one primary loader must be active at all times.
  • On-demand loader: High-specification node for large schemas and heavy workloads. Starts automatically when a load plan that references it is triggered and shuts down after a configurable idle period.
    Important

    The on-demand loader must be configured as a self-managed cluster. Standard multi-loader configurations are not supported.

You assign specific load plans to the on-demand loader. All other load plans and schema update jobs continue to run on the primary loader without change.

First group requirement

The on-demand loader must be assigned to the first load group of the load plan for automatic startup to trigger. If an on-demand loader is assigned to a group in the middle of a load plan, it will not be started automatically; the job will be assigned to one of the available loaders instead.

When to use the on-demand loader

The on-demand loader is most effective when your deployment has all of the following characteristics:

  • You are running on a Cloud deployment with a self-managed loader cluster.
  • You have one or more load plans that require significantly more CPU and memory than your typical workload.
  • Those heavy load plans execute infrequently, for example, once or a few times per day.
  • Your always-on loader is sized appropriately for lighter workloads and does not need to be high-specification for the majority of the day.
Important

The on-demand loader is not suitable for load plans that run continuously throughout the day.

On-demand loader lifecycle

Important: Do not start or stop the on-demand loader manually from the CMC

Starting or stopping the on-demand loader from the CMC stops the Incorta service but does not shut down the underlying node, bypassing the cost-saving lifecycle management and preventing the automatic startup mechanism from functioning correctly.

Start up for a manually executed load plan

When you manually start a load plan execution, and the first load group is assigned to an on-demand loader that is currently shut down, the Analytics Service automatically calls the cloud API to start the on-demand loader node. The load plan moves to a Loader spin-up status and waits until the loader is available. Once the loader is ready, the load plan proceeds normally.

Note

The Loader spin-up timeout is configurable. If the loader does not become available within the configured timeout (default: 1 hour), the load plan fails.

Pre-start for scheduled jobs

A background process on the Analytics Service monitors upcoming scheduled load plans. If a scheduled load plan is assigned to an on-demand loader that is currently shut down, the Analytics Service starts the loader in advance of the scheduled execution time. The pre-start window is configurable (default: 1 hour before execution).

Shutdown

If the on-demand loader has no active jobs and no upcoming scheduled jobs within the pre-start window, and remains idle for a configurable period (default: 1 hour), the Analytics Service calls the cloud API to shut down the node. The shutdown terminates the entire node, not just the Incorta service, to achieve full infrastructure cost savings.

Loader election rules

The on-demand loader participates in job assignment only in specific cases.

Elected forNot elected for

  ●  Load jobs of a load plan where the first load group is explicitly assigned to the on-demand loader
  ●  Manual schema executions in a load group that has the on-demand loader assigned
  ●  Schema update jobs
  ●  Load jobs that have not been explicitly assigned to it

Behavior with multiple assigned loaders

If the first load group has multiple loaders assigned (including an on-demand loader):

  • If any of the assigned loaders is already running, no startup is triggered — the available loader handles the job.
  • If all assigned loaders are shut down, the Analytics Service starts the first on-demand loader in the list.---