1. Home
  2. Latest Release Docs
  3. Guides
  4. Concepts
    1. Aggregate filter
    2. Alias
    3. Apache Avro™
    4. Apache Kafka™
    5. Applied filter
    6. Base table
    7. Bookmark
    8. Business schema
    9. Business schema view
    10. Business schema view column
    11. Business schema view formula column
    12. Column
    13. Connector
    14. Dashboard filter
    15. Dashboard folder
    16. Dashboard runtime filter
    17. Data agent
    18. Data alert
    19. Data destination
    20. Data purge
    21. Data retention
    22. Date system variable
    23. Distinct filter
    24. Dynamic field
    25. Expression
    26. External data source
    27. External session variable
    28. Filter expression
    29. Filter expression session variable
    30. Filter option
    31. Formula column
    32. Formula expression
    33. Full load
    34. Global variable
    35. Incorta Analyzer Table
    36. Incorta Analyzer Table column
    37. Incorta Analyzer Table formula column
    38. Incorta Analyzer View
    39. Incorta Analyzer View formula column
    40. Incorta Spark SQL View
    41. Incorta SQL Table
    42. Individual filter
    43. Insight filter
    44. Internal query expression
    45. Internal session variable
    46. Join
    47. Join condition
    48. Join condition column
    49. Join filter
    50. Join filter column
    51. Level-based measure
    52. Load filter
    53. Local data file
    54. Local data folder
    55. Materialized View
    56. Materialized View column
    57. Materialized View formula column
    58. Measure filter
    59. Miscellaneous system variable
    60. Multi-source table
    61. Physical schema
    62. Physical schema table
    63. Physical schema table column
    64. Physical schema table formula column
    65. Physical schema table runtime security filter
    66. PostgreSQL (Incorta SQL) View
    67. Presentation variable
    68. Prompt
    69. Query system variable
    70. Runtime business view
    71. Runtime business view formula column
    72. Runtime security filter
    73. Schema evolution
    74. Session variable
    75. Shared Storage
    76. Upgrade Readiness
    77. Variable
    78. Visualization
  5. Tools
  6. References
  7. Integrations
  8. Data applications
  9. Release Notes
  10. Incorta Premium

Concepts → Data Retention

Overview

Data retention refers to the practice of maintaining data for a specific period. Starting 2024.7.0, Incorta has introduced new table-level settings for schema managers to better control the data stored on disk based on predefined criteria. This feature allows schema managers to specify criteria for keeping or deleting data on the shared storage, promoting efficient data management, improving performance, and optimizing resource and disk space usage.

Data retention policies can apply only to physical schema tables and materialized views (MVs). A purge job is required to remove data that does not meet the retention criteria.

Creating a data retention policy

Data retention settings can be configured on the Advanced Settings tab of the Table Editor for any physical table or MV. You can define data retention policies using time-window configurations or custom conditions.

Warning

Exercise caution when setting criteria for data retention. Once data is purged, it is irretrievable. You can fully load affected tables and MVs to recover from an accidental purge operation.

Data retention via a time-window configuration

For time-window retention policies, you specify the time window based on a date or timestamp column in the table or MV. Records within the defined time window will be retained, while those outside will be marked for deletion during the next purge job.

Data retention via a custom condition

You can also create a custom condition that defines which data to retain. Records satisfying the custom condition will remain while those not satisfying the condition will be marked for deletion.

Custom conditions offer more flexibility than time-window configurations. Within a custom condition, you can:

  • Reference columns of different data types
  • Use system variables
  • Use different types of built-in functions
  • Specify more complex conditions

Purging unneeded data

After configuring a data retention policy, data that does not meet the retention criteria can be removed via a data purge job. You can execute a purge job manually for individual tables or MVs or across all tables and MVs within a schema in the same dialogues you use to perform other load actions. Alternatively, you can schedule purge jobs (via a load plan) to clean up data from one or more physical schemas.

Notes

The purge job creates a new version of table data (Parquet files). These files may have records in a different order compared to the previous versions.

For more details about data purge jobs, refer to Concepts → Data Purge.