1. Home
  2. Previous Releases
  3. Latest Release
  4. Guides
  5. Concepts
    1. Aggregate filter
    2. Alias
    3. Apache Avro™
    4. Apache Kafka™
    5. Applied filter
    6. Base table
    7. Bookmark
    8. Business schema
    9. Business schema view
    10. Business schema view column
    11. Business schema view formula column
    12. Column
    13. Connector
    14. Dashboard filter
    15. Dashboard folder
    16. Dashboard runtime filter
    17. Data agent
    18. Data alert
    19. Data destination
    20. Data Purge
    21. Data Retention
    22. Date system variable
    23. Distinct filter
    24. Dynamic Field
    25. Expression
    26. External data source
    27. External session variable
    28. Filter expression
    29. Filter expression session variable
    30. Filter option
    31. Formula column
    32. Formula expression
    33. Full load
    34. Global variable
    35. Incorta Analyzer Table
    36. Incorta Analyzer Table column
    37. Incorta Analyzer Table formula column
    38. Incorta Analyzer View
    39. Incorta Analyzer View formula column
    40. Incorta Spark SQL View
    41. Incorta SQL Table
    42. Individual filter
    43. Insight filter
    44. Internal query expression
    45. Internal session variable
    46. Join
    47. Join condition
    48. Join condition column
    49. Join filter
    50. Join filter column
    51. Level based measure
    52. Load filter
    53. Local data file
    54. Local data folder
    55. Materialized View
    56. Materialized View Column
    57. Materialized View Formula Column
    58. Measure filter
    59. Miscellaneous system variable
    60. Multi-Source table
    61. Physical schema
    62. Physical schema table
    63. Physical schema table column
    64. Physical schema table formula column
    65. Physical schema table runtime security filter
    66. PostgreSQL (Incorta SQL) View
    67. Presentation variable
    68. Prompt
    69. Query system variable
    70. Runtime business view
    71. Runtime business view formula column
    72. Runtime security filter
    73. Schema evolution
    74. Session variable
    75. Shared Storage
    76. Upgrade Readiness
    77. Variable
    78. Visualization
  6. Tools
  7. References
  8. Integrations
  9. Data applications
  10. Releases
  11. Incorta Premium

Concepts → Data Purge

Overview

Data Purge is a new load type introduced in 2024.7.x to remove unnecessary data from physical tables or materialized views (MVs). This process enhances system performance while optimizing disk space usage.

Prepare for a data purge

Configure objects

Before executing a data purge job, you must configure the relevant objects and identify the data to be deleted. Unneeded data can be designated per object through a data retention policy or by defining an exclusion set.

  • A data retention policy defines the conditions data must meet to be retained; otherwise, it is marked for deletion.
  • Alternatively, an exclusion set is a physical schema object containing the identifiers of rows designated for deletion. You must specify the mapping columns in the target table and its corresponding exclusion set.

Configure Spark

The purge process is run via Spark. You can enable dynamic resource allocation for Spark applications running data purge and Parquet merge processes, ensuring efficient resource usage based on the workload demands.

Spark application configurations can also be customized, allowing for tailored behavior to meet specific operational requirements. Parameters such as application cores, memory for executors, and driver memory can be adjusted in the Cluster Management Console (CMC) > Server Configurations > Spark Integration.

Start or schedule a data purge job

A Super User or any user with the SuperRole or Schema Manager roles can initiate a data purge job or schedule one or more unattended purge jobs for multiple physical schemas within a load plan.

You can start or schedule a purge job from the Schema Designer. You can also create a data purge scheduled plan from the Schema Manager or the Scheduler.

The data purge load type is available for:

The data purge cycle

When a data purge job is executed for one or more schemas, all schema objects (excluding aliases) are included in the load job. However, only physical tables and MVs with data purge configurations have their data processed to be deleted while other objects are loaded from staging.

Objects with data retention policies

Incorta reads and evaluates records against the data retention condition to get the list of records to delete for each Parquet file.

  • If records need to be deleted,
    • Incorta sends the data purge script to Spark, which includes the path to the target table and a list of Parquet files containing the records marked for deletion.
    • Spark deletes data that does not meet the retention policies and creates a new version of the Parquet files and incremental segments.
  • If no records need to be deleted, Incorta does not submit a job to Spark and performs a regular load-from-staging job, in which no new file version is created.

Objects with a deletion dataset

Incorta sends the data purge script with the following details to Spark:

  • The path to the compacted version of the exclusion table
  • The mapping columns in the target object and the exclusion table

Spark deletes data shared between the source tables and their exclusion sets and creates a new version of the Parquet files and incremental segments. Spark will create a new version even if no records have been deleted.

Objects with data retention and deletion dataset configurations

  1. Incorta decides on whether the data should be deleted per the data retention policy.
  2. Incorta sends the data purge script with the details required for both the data retention condition and the exclusion dataset.
  3. Spark executes the script, deletes the required data, and creates a new version of Parquet files and segments. If Spark fails to execute the purge script for any of the two configurations:
    • No data is deleted.
    • No new version is written.
    • The load of the table or MV continues as a staging load.

Additional considerations

  • If an exclusion dataset has data purge configurations, it can’t be in the same load job as its target table.
  • The new Parquet files written by the data purge step have shuffled data.
  • Only physical tables and MVs can be used as exclusion datasets.
  • Only data columns can be used when defining an exclusion dataset.