Guides → Migrate

Schema Developers, Dashboard Developers, Tenant Super Users, System Administrators, and a Cluster Management Console (CMC) Administrator migrate specified objects of a tenant, entire tenants, host files, and CMC configurations to ensure that tenant functionality is maintained between environments.

Types of Environments

Each environment represents a stage in the software development life cycle, from development to production. Migration centers around how to move content from a development stage to production. Dashboards, physical schemas, business schemas, session variables, external data sources, users and groups, tenant configurations, server configurations, and system configuration file are examples of migratable content.

For enterprise deployments of Incorta, there are typically three environments with each environment representing its own stage:

Development environment

Schema and dashboard developers create new content in the development (DEV) environment. The DEV environment represents a desired ideal for the business community in production. The DEV environment often has a subset copy of production data. Developers safely explore, learn, demo, develop, and unit test new content in this environment.

User Acceptance Testing environment

Functional experts, business users, and quality assurance engineers perform tests in the user acceptance testing (UAT) environment. Functional experts such as product owners, project managers and program managers, oversee the technical side of development and help ensure that specifications are met. Quality assurance engineers administer performance tests to measure optimizations. Business users know exactly what the outcome should look like. Together, these users test content in UAT to determine its acceptance for production. UAT may have a subset or copy of production data.

Production environment

Having passed UAT, new content migrates to production in the production (PROD) environment. Administrators and Super Users migrate content from UAT to PROD. In production, ad-hoc changes are often highly regulated or prohibited. Real-world users are dashboard consumers. Dashboard consumers personalize, filter, bookmark, and share dashboards in production.

Recommended Environment Configuration

There are ideal configurations for each of the deployment environments. When you use and maintain these configurations, you will mitigate failures during migration.

Incorta Version

You should maintain the same Incorta version across all environments. This guide will assume that the Incorta version for all of your environments is the same.

Migration Path

The ideal environment migration path is DEV → UAT → PROD. It is important to maintain consistent functionality between environments. There may be use cases where it is necessary to migrate changes from PROD back to UAT and DEV, but these cases should be minimal. An example use case is a dashboard change made directly in PROD that is migrated back to UAT and DEV. Permissible operations by environment and user role may differ by organization, and require that you establish clear change management guidelines.

A change management process should also require the following for every change:

• A test case, proven test results, and user acceptance
• An impact analysis, or comparison between the UAT and PROD environments

Tenant Backups

You should export a tenant backup in the DEV, UAT and PROD environments on a regular basis. You can schedule these backups to occur with a utility such as crontab. Tenant backups allow you to revert to an older, working version of the environment in the event that an issue is introduced during development or migration.

Group and User Permissions

User and group permissions are a useful way to mitigate unplanned changes in the PROD tenant and maintain data security.

In the DEV and UAT environments, group level permissions should be used to share content. Since permissions are not migrated along with content, it is simpler to manage group level permissions than individual level permissions.

In the PROD environment, primarily Administrators or Super Users should have Can Edit permissions. User accessibility use cases should be tested in the UAT environment before migrating to the PROD environment. Restrict edit controls in the PROD environment to encourage the practice of changing the Incorta tenant in DEV, and migrating the changes to UAT and PROD.

Prerequisites for Migration

Run the Inspector Tool

The Inspector tool checks the lineage references of Incorta metadata objects including tables, schemas, business schemas, business schema views, dashboards, and session variables. For the purposes of migration, the CMC administrator should run the Inspector tool before and after a migration. The Inspector Tool will identify broken references, unused objects, and other issues that may result from a migration. Refer to the Inspector Tool documentation for additional information on how to run the tool and view results.

Run the Formula Validation Tool

The Formula Validation tool identifies issues with the formula expression syntax in dashboards, business schemas, and schemas. In order to validate the formula expression syntax, the tool requires a tenant export file. The CMC Administrator can create the tenant export using the Cluster Management Console (CMC).

Types of Migration

Migrate as a Developer

Developers typically migrate specific tenant objects, which is the simplest form of migration. Ideally use specified object migration in late phase development or in a continuous integration continuous deployment (CICD), or similar process. Also use it in early phase development if the DEV environment has a large number of extraneous objects not relevant to the UAT or PROD environments. The following are the tenant objects that can be independently moved from one environment to another, and the migration steps.

Migrate External Data Sources

You need to manually create external data sources in each environment you move to.

Migrate Schemas

Migrate schemas with the Incorta Direct Data Platform™ user interface. Here are the schema migration steps in the Incorta Direct Data Platform™:

• In the Navigation bar of the source environment, select Schema.
• Select the checkbox to the left of the schema you would like to export.
• In the Action menu that appears (⋮ vertical ellipsis icon), select Export.
• In the Navigation bar of the target environment, select Schema.
• In the Action bar, select select + New → Import Schema.
• Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing schema checkbox if you are importing a schema with the same name as an existing schema.
• The Import Results dialog will contain a status of Created if the import was successful. Select Close.

Migrate Business Schemas

Migrate business schemas with the Incorta Direct Data Platform™ user interface. Here are the business schema migration steps in the Incorta Direct Data Platform™:

• In the Navigation bar of the source environment, select Business Schema.
• Select the checkbox to the left of the business schema you would like to export.
• In the Action menu that appears (⋮ vertical ellipsis icon), select Export.
• In the Navigation bar of the target environment, select Business Schema.
• In the Action bar, select select + New → Import Business Schema.
• Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing business schema checkbox if you are importing a business schema with the same name as an existing business schema.
• The Import Results dialog will contain a status of Created if the import was successful. Select Close.

Migrate Schema Load Jobs

Migrate schema load jobs manually from one environment to another. You can recreate the schema load in the target environment based on the load job configuration in the source environment. Refer to the Scheduler documentation for additional information.

Migrate Schema Alerts

Migrate schema alerts manually from one environment to another. You can recreate the schema notification in the target environment based on the schema notification configuration in the source environment. Refer to the Scheduler documentation for additional information.

Migrate Dashboards

Migrate dashboards with the Incorta Direct Data Platform™ user interface. Here are the dashboard migration steps in the Incorta Direct Data Platform™:

• In the Navigation bar of the source environment, select Content.
• Select the dashboard you would like to migrate to open it.
• In the More Options menu (⋮ vertical ellipsis icon), select Export.
• In the Export Dashboard dialog, select Export.
• In the Navigation bar of the target environment, select Content.
• In the Action bar, select select + New → Import Folder/Dashboard.
• Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing dashboard checkbox if you are importing a dashboard with the same name as an existing dashboard.
• The Import Results dialog will contain a status of Created if the import was successful. Select Close.

Migrate Dashboard Alerts

Migrate dashboard alerts with the Incorta Direct Data Platform™ user interface. Here are the Dashboard Alert migration steps in the Incorta Direct Data Platform™:

• In the Navigation bar of the source environment, select Scheduler, and then the Data Notifications tab.
• Select the checkbox to the left of the dashboard alert you would like to export.
• In the Action menu that appears, select Export (Download icon).
• In the Navigation bar of the target environment, select Scheduler, and then the Data Notifications tab.
• In the Action bar, select select + New → Import Notifications.
• Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing alerts checkbox if you are importing a data alert with the same name as an existing data alert.
• The Import Results dialog will contain a status of Created if the import was successful. Select Close.

Migrate Dashboard Delivery Jobs

Migrate dashboard delivery jobs with the Incorta Direct Data Platform™ user interface when you migrate the associated dashboard. Select the Include Scheduled Jobs checkbox in the Export dialog.

Migrate Bookmarks

Migrate bookmarks with the Incorta Direct Data Platform™ user interface when you migrate the associated dashboard. Select the Include Bookmarks checkbox in the Export dialog. Both public and private bookmarks will be preserved.

Migrate Global Variables

Migrate global variables manually from one environment to another. You can recreate the global variable in the target environment based on the global variable configuration in the source environment. Refer to the Schema Manager documentation for additional information.

Migrate Session Variables

You must include Session variables while migrating a tenant, otherwise you will need to recreate them manually in the other environment. For more assistance, please contact the Cloud support team.

Migrate Groups

Migrate groups manually from one environment to another. You can recreate the groups in the target environment based on the group configuration in the source environment. Refer to the User Profile Manager documentation for additional information.

Migrate Users

Migrate users manually from one environment to another. You can recreate the users in the target environment based on the user configuration in the source environment. Refer to the User Profile Manager documentation for additional information.

A more complex scenario involves the migration of more than one of the tenant objects listed above. It is critical to ensure that dependent objects are not broken when a change is made to a tenant object. For example, when you change a business schema column, it is important to migrate the affected dashboards. As mentioned earlier, the Inspector Tool will help to identify any migration issues.

Migrate as a Tenant Super User

Super Users typically migrate full tenants. Ideally use full tenant migration in early phase development for the initial creation of the UAT or PROD environments. The following are considerations when you migrate a full tenant:

• All objects are migrated with the original ownership. It is recommended that the objects in PROD are owned by Administrators or Super Users, not by developers.
• Data sources point to databases in DEV or UAT that are different than in PROD.
• There is not an option to exclude certain objects from migration.

Migrating a full tenant using CMC:

Before you migrate the full tenant, it is important to first create a tenant backup.

Backup the Existing Tenant in the Target Environment

The following are the steps to create a tenant backup using the CMC:

• Sign in to the CMC of the source environment.
• In the Navigation bar, select Clusters.
• In the cluster list, select a Cluster name.
• In the canvas tabs, select Tenants.
• In the More Options menu (⋮ vertical ellipsis icon) at the end of the tenant row, select Backup.

Use the CMC to Migrate the Full Tenant

The following are the steps to migrate the full tenant using the CMC:

• Sign in to the CMC of the source environment.

• In the Navigation bar, select Clusters.

• In the cluster list, select a Cluster name.

• In the canvas tabs, select Tenants.

• In the More Options menu (⋮ vertical ellipsis icon) on the right side of the tenant row, select Execute inspector now to review and verify the tenant object lineage.

• In the More Options menu (⋮ vertical ellipsis icon) on the right side of the tenant row, select Export. The tenant will be packaged into a tenant_<TENANT_NAME>.zip folder containing:

  • <TENANT_NAME>.properties file
  • <TENANT_NAME>.zip file
    • dashboards folder containing XML
    • datasources folder containing XML
    • schemas folder containing XML
    • tenant.xml file

• The exported ZIP file will be downloaded to your browser’s default download folder.

• Sign in to the CMC of the target environment.

• In the Navigation bar, select Clusters.

• In the cluster list, select a Cluster name.

• In the canvas tabs, select Tenants.

• Select + → Import Tenant.

• Drag and drop the tenant_<TENANT_NAME>.zip to the Click or drag a file here to upload panel in the Import a tenant to the cluster dialog.

• Select Next.

• If there is a <TENANT_NAME>.properties file located within tenant_<TENANT_NAME>.zip, select Yes, Use it to autofill the tenant property values. The tenant property values can be changed later. Otherwise, select No, ignore it.

• Verify or enter the tenant properties:

  Property	Control	Description
  Name	text box	Enter the tenant name. Select Check to determine if a Tenant already exists with the name entered.
  Username	text box	Enter the username for the Super User
  Password	text box	Enter the password for the Super User
  Email	text box	Enter the email address for the Super User
  Path	text box	Enter the shared storage path for tenant related data
  Pause scheduled jobs	toggle	Enable this property if the imported tenant will have all scheduled jobs paused on import

• Enter and verify the tenant email properties:

  Property	Control	Description
  Sender’s Username Auth	toggle	Enable this property if the email requires username authentication
  System Email Username	text box	Enable Sender’s Username Auth to configure this property. Enter the username for the system email.
  System Email Address	text box	Enter the email address for the system email
  System Email Password	text box	Enter the password for the system email address
  SMTP Host	text box	Enter the Simple Mail Transfer Protocol (SMTP) host for the system email
  SMTP Port	text box	Enter the SMTP port number
  Share Notifications	toggle	Enable this property to share notifications

• Select Create to finalize the tenant migration.

  Important

  You may need to enter the username and password for some of your imported connectors that are in the tenant. This affects the following connectors:

  • Amazon Web Services (AWS)
  • RedShift
  • Apache Drill
  • Apache Hive
  • Cassandra (Simba)
  • Custom SQL
  • IBM DB2
  • IBM Netezza
  • Microsoft SQL Server
  • Microsoft SQL Server (JTDS)
  • MongoDB BI
  • MySQL
  • NetSuite - SuiteAnalytics
  • Oracle
  • Presto
  • PostgreSQL
  • SAP Hana
  • SAP Sybase IQ
  • Teradata
  • Vertica

• Run the Inspector Tool following the migration to identify any lineage errors that may have occurred from moving the tenant between environments.

Migrate as a CMC Administrator

CMC Administrators typically migrate CMC configurations. CMC configurations are migrated manually from one environment to another. You can replicate the CMC configurations in the target environment based on the CMC configurations in the source environment. Adjust any configuration values as necessary for the target environment. The following are the CMC configurations to consider for migration.

Migrate General Configurations

Migrate Node Services Configurations

Migrate the On Heap Memory (GB), Off Heap Memory (GB), and CPU Utilization (%) for both the Analytic and Loader services as follows:

• Sign in to the CMC.
• In the Navigation bar, select Nodes.
• In the Services tab, select either an Analytics or Loader Service.
• Select Edit (Pen icon).
• Update the node services configurations as necessary.
• Select Update.

Migrate Notebook Add-on

Enable Notebook Add-on for for the Analytics service as follows:

• Sign in to the CMC.
• In the Navigation bar, select Nodes.
• Select the Add-ons tab.
• Select + → Add Notebook.

Migrate Server Cluster Configurations

Migrate the server cluster configurations as follows:

• Sign in to the CMC.
• In the Navigation bar, select Clusters.
• In the cluster list, select a Cluster name.
• In the canvas tabs, select Cluster Configurations.
• In the panel tabs, select Server Configurations.
• In the left pane, select the one of the following options to update the associated server configurations:
  • Kafka Consumer Service Name
  • SQL Interface
  • Spark Integration
  • Tuning
  • UI Customizations
  • Diagnostics

Migrate Default Tenant Cluster Configurations

Migrate the default tenant configurations as follows:

• Sign in to the CMC.
• In the Navigation bar, select Clusters.
• In the cluster list, select a Cluster name.
• In the canvas tabs, select Cluster Configurations.
• In the panel tabs, select Default Tenant Configurations.
• In the left pane, select one of the following options to update the associated default tenant configurations:
  • Security - Authentication Type
  • Regional Settings
  • Email - SMTP Host, Enable SMTP SSL, Enable Notifications, Enable Internal Error Notifications
  • Data Loading - Stop Loading on First Error, Pause Scheduled Jobs
  • Integration - Google Maps API Key, Apple Maps Developer Team ID, Apple MapKit JS Key ID, Apple Maps API Key, Google Drive Client ID, Google Drive Client Secret, Dropbox App Key, Dropbox App Secret, Box Client ID, Box Client Secret, Mapbox API KeyTenant EFS
  • Advanced - Insight Max Groups UI Default, Insight Max Groups Limit, Insight Max Values in Filter Subquery, Force Reload Columns, Sync In Background, Warmup Mode
  • Tuning - Schema Pool Size/Maximum Concurrent Load Jobs, Table Maximum Parallel Chunks, Evaluate Session Variables At Login
  • Incorta Labs - Enable Insight 'View As' Menu, Notebook Integration
  • External Visualization - Incorta Host, Incorta Port