1. Latest Release Home
  2. Guides
  3. Concepts
  4. Tools
    1. AI App Manager
    2. Alert Editor
    3. Alias Sync Tool
    4. Analyzer
    5. Apache Parquet Merge Tool
    6. Avro Extractor Tool
    7. Business Schema Designer
    8. Business Schema Manager
    9. Cluster Management Console
    10. CMC Logs Manager
    11. CMC Monitoring
    12. CMC Patch Manager
    13. CMC Scheduler
    14. CMC Tenant Manager
    15. Command Line Interface
    16. Content Manager
    17. Dashboard Filters Manager
    18. Dashboard Manager
    19. Data Agent
    20. Data APIs for External Notebooks
    21. Data Lineage Viewer v2
    22. Data Manager
    23. Data Studio Manager
    24. Formula Builder
    25. Inspector Tool
    26. Join Editor
    27. Load Job Details Viewer
    28. Load Plan DAG Viewer
    29. Model Update Viewer
    30. Notebook Editor
    31. Profile Manager
    32. Public API
    33. Query Plan Viewer
    34. Scheduler
    35. Schema Designer
    36. Schema Diagram Viewer
    37. Schema Manager
    38. Schema Version Compare
    39. Schema Wizard
    40. Security Manager
    41. Spark Issue Detector
    42. Table Editor
    43. Tenant Management Tool
    44. Upload Service Command Line Interface (CLI)
    45. Versioning Migration Tool
  5. References
  6. Integrations
  7. Data applications
  8. Data Governance
  9. Release Notes
  10. Incorta Premium
  11. Release Support Policy
  12. Release Model

Tools → Spark Issue Detector

Overview

The Spark Issue Detector tool was originally released and used during the upgrade from Spark 2 to Spark 3. Upgrading to Incorta v8.0.0 - 2026.8 upgrades Spark to 4.0.2 and Python to 3.12. In this Incorta release, the tool has been updated to detect potential compatibility issues related to the Spark 4 and Python 3.12 upgrade. The Spark Issue Detector tool helps identify potential compatibility issues before you upgrade to Incorta v8.0.0 - 2026.8.

Important

This tool performs static analysis only and generates a report. It does not fix issues automatically. It scans materialized view (MV) scripts, MV-level Spark properties, and cluster-wide Spark configurations for patterns that may be incompatible with the target Spark/Python version.

Reported findings are potential concerns, not guaranteed failures. Review each finding to determine whether action is needed.

What the tool checks

  • Materialized view (MV) scripts
  • MV-level Spark properties
  • Cluster-wide Spark configuration (optional, requires CMC credentials)

Prerequisites

  • A tenant export of the environment you want to upgrade.
  • Shell access to run issue-detector.sh.
  • (Optional, but recommended) CMC URL and credentials, if you want the tool to check the global Spark configurations.

Step 1: Take a tenant export

Generate a tenant export of the environment you plan to upgrade. The tool uses this export to analyze MV scripts and Spark-related settings.

Step 2: Run the Issue Detector

Run the script against the tenant export path:

./issue-detector.sh -t=TENANT_EXPORT_PATH [OPTIONS]

Available options

OptionDescription
-t(Required) Path to the tenant export
-s, --current-versionCurrent Spark version in use (major.minor or major.minor.patch). Defaults to 3.3.0
-n, --target-versionTarget Spark version (major.minor or major.minor.patch). Defaults to 4.0.2
-o, --output-dirOutput directory for the generated report and issue reference
--incorta-cmc-urlBase URL of the Incorta CMC / server (used to fetch the global Spark configuration)
--incorta-cmc-usernameUsername to log in to the Incorta CMC / server
--incorta-cmc-passwordPassword to log in to the Incorta CMC / server
--skip-global-config-checkExplicitly skip the global Spark configuration check
--cluster-nameCluster name; suppresses findings the cluster already satisfies (e.g., Pandas/NumPy version checks)
-h, --helpPrint help options
Note

You must do one of the following:

  • Provide --incorta-cmc-url, --incorta-cmc-username, and --incorta-cmc-password to check the global Spark configurations.
  • Explicitly add --skip-global-config-check to skip that check.

The script will not proceed with the global configuration check silently skipped; one of these two paths must be chosen.

Example: Full check (Upgrading from Spark 3.3.0 to Spark 4.0.2, with CMC check)

./issue-detector.sh -t=/path/to/tenant_export
  -s=3.3.0
  -n=4.0.2
  -o=/path/to/output
  --incorta-cmc-url=https://your-cmc-host
  --incorta-cmc-username=admin
  --incorta-cmc-password=your_password

Example: Skipping the global configuration check

./issue-detector.sh -t=/path/to/tenant_export
  -o=/path/to/output
  --skip-global-config-check

Step 3: Review the report

Once the script finishes, check the output directory for the generated report and issue reference. For each finding:

  1. Identify the affected MV script, property, or configuration.
  2. Cross-reference the finding with the issue reference documentation included in the output.
  3. Determine whether the finding requires action. Remember that the tool performs static analysis, so a flagged item does not necessarily indicate a runtime failure.

Step 4: Address applicable findings before upgrading

Review and address all findings that require action before proceeding with the upgrade. If you're unsure how to address a specific finding, contact Incorta Support for upgrade assistance.

Additional post-upgrade step: External Notebooks

If you use external Notebooks with the data APIs for external Notebooks, note that the previously used Python library is not compatible with Spark 4.

After upgrading to v8.0.0 - 2026.8, reinstall the Python library using:

pip install IncortaAnalytics/IncortaNode/bin/data_apis/python/incorta_data_apis-1.0-py3-none-any.whl