1. Home
  2. Previous Releases
  3. Latest Release
  4. Guides
  5. Concepts
    1. Aggregate filter
    2. Alias
    3. Apache Avro™
    4. Apache Kafka™
    5. Applied filter
    6. Base table
    7. Bookmark
    8. Business schema
    9. Business schema view
    10. Business schema view column
    11. Business schema view formula column
    12. Column
    13. Connector
    14. Dashboard filter
    15. Dashboard folder
    16. Dashboard runtime filter
    17. Data agent
    18. Data alert
    19. Data destination
    20. Data Purge
    21. Data Retention
    22. Date system variable
    23. Distinct filter
    24. Dynamic Field
    25. Expression
    26. External data source
    27. External session variable
    28. Filter expression
    29. Filter expression session variable
    30. Filter option
    31. Formula column
    32. Formula expression
    33. Full load
    34. Global variable
    35. Incorta Analyzer Table
    36. Incorta Analyzer Table column
    37. Incorta Analyzer Table formula column
    38. Incorta Analyzer View
    39. Incorta Analyzer View formula column
    40. Incorta Spark SQL View
    41. Incorta SQL Table
    42. Individual filter
    43. Insight filter
    44. Internal query expression
    45. Internal session variable
    46. Join
    47. Join condition
    48. Join condition column
    49. Join filter
    50. Join filter column
    51. Level based measure
    52. Load filter
    53. Local data file
    54. Local data folder
    55. Materialized View
    56. Materialized View Column
    57. Materialized View Formula Column
    58. Measure filter
    59. Miscellaneous system variable
    60. Multi-Source table
    61. Physical schema
    62. Physical schema table
    63. Physical schema table column
    64. Physical schema table formula column
    65. Physical schema table runtime security filter
    66. PostgreSQL (Incorta SQL) View
    67. Presentation variable
    68. Prompt
    69. Query system variable
    70. Runtime business view
    71. Runtime business view formula column
    72. Runtime security filter
    73. Schema evolution
    74. Session variable
    75. Shared Storage
    76. Upgrade Readiness
    77. Variable
    78. Visualization
  6. Tools
  7. References
  8. Integrations
  9. Data applications
  10. Releases
  11. Incorta Premium

Concepts → Schema Evolution

Overview

Before 2024.7.x, adding and removing physical columns required performing a full load of the respective physical table or materialized views (MVs). Starting 2024.7.x, Incorta has leveraged Delta Lake column mapping to support schema evolution, streamlining adding and removing columns without performing a full load of the respective objects.

Upgrade considerations

  • As a result of using Delta Lake column mapping, Incorta has modified the definition of columns in the schema metadata. After upgrading to 2024.7.x or later releases, the column definition will change with the next schema update.

  • When importing schemas from previous releases with the overwrite option, you can keep existing schema data. When the Keep Data option is selected,

    • The imported schema XML is updated with the new table and column definitions.
    • Incorta can use existing Parquet files.
    • You do not need to perform a full load of the imported schema.

Column name considerations

Column names in Incorta are case-sensitive. Renaming a column (in the extraction query, for example) or changing its letter case results in a new column and typically does not require a full load starting 2024.7.x. Reverting the name changes results in a new column as well.

Notes
  • The following actions still require a full load of the respective object:
    • Adding, changing, or removing a key column
    • Encrypting a column
    • Changing the column data type
  • You must revalidate the query of MVs referencing removed or updated columns.