Guides → Migrate
Schema Developers, Dashboard Developers, Tenant Super Users, System Administrators, and a Cluster Management Console (CMC) Administrator migrate specified objects of a tenant, entire tenants, host files, and CMC configurations to ensure that tenant functionality is maintained between environments.
Types of Environments
Each environment represents a stage in the software development life cycle, from development to production. Migration centers around how to move content from a development stage to production. Dashboards, physical schemas, business schemas, session variables, external data sources, users and groups, tenant configurations, server configurations, and system configuration file are examples of migratable content.
For enterprise deployments of Incorta, there are typically three environments with each environment representing its own stage:
Development and the DEV environment
Schema and dashboard developers create new content in the DEV environment. The DEV environment represents a desired ideal for the business community in production. The DEV environment often has a subset copy of production data. Developers safely explore, learn, demo, develop, and unit test new content in this environment.
User Acceptance Testing and the UAT environment
Functional experts, business users, and quality assurance engineers perform tests in the UAT environment. Functional experts such as product owners, project managers and program managers, oversee the technical side of development and help ensure that specifications are met. Quality assurance engineers administer performance tests to measure optimizations. Business users know exactly what the outcome should look like. Together, these users test content in UAT to determine its acceptance for production. UAT may have a subset or copy of production data.
Production and the PROD environment
Having passed UAT, new content migrates to production in the PROD environment. Administrators and Super Users migrate content from UAT to PROD. In production, ad-hoc changes are often highly regulated or prohibited. Real-world users are dashboard consumers. Dashboard consumers personalize, filter, bookmark, and share dashboards in production.
Recommended Environment Configuration
There are ideal configurations for each of the deployment environments. When you use and maintain these configurations, you will mitigate failures during migration.
Incorta Version
You should maintain the same Incorta version across all environments. This guide will assume that the Incorta version for all of your environments is the same.
Migration Path
The ideal environment migration path is DEV → UAT → PROD. It is important to maintain consistent functionality between environments. There may be use cases where it is necessary to migrate changes from PROD back to UAT and DEV, but these cases should be minimal. An example use case is a dashboard change made directly in PROD that is migrated back to UAT and DEV. Permissible operations by environment and user role may differ by organization, and require that you establish clear change management guidelines.
A change management process should also require the following for every change:
- A test case, proven test results, and user acceptance
- An impact analysis, or comparison between the UAT and PROD environments
Tenant Backups
You should export a tenant backup in the DEV, UAT and PROD environments on a regular basis. You can schedule these backups to occur with a utility such as crontab. Tenant backups allow you to revert to an older, working version of the environment in the event that an issue is introduced during development or migration.
Group and User Permissions
User and group permissions are a useful way to mitigate unplanned changes in the PROD tenant and maintain data security.
In the DEV and UAT environments, group level permissions should be used to share content. Since permissions are not migrated along with content, it is simpler to manage group level permissions than individual level permissions.
In the PROD environment, primarily Administrators or Super Users should have Can Edit permissions. User accessibility use cases should be tested in the UAT environment before migrating to the PROD environment. Restrict edit controls in the PROD environment to encourage the practice of changing the Incorta tenant in DEV, and migrating the changes to UAT and PROD.
If you are using Lightweight Directory Access Protocol (LDAP) or Single Sign-On (SSO), group and user integration will need to be performed on the UAT and PROD environments. Refer to Secure Login Access for LDAP and SSO configurations.
Prerequisites for Migration
Run the Inspector Tool
The Inspector tool checks the lineage references of Incorta metadata objects including tables, schemas, business schemas, business schema views, dashboards, and session variables. For the purposes of migration, the CMC administrator should run the Inspector tool before and after a migration. The Inspector Tool will identify broken references, unused objects, and other issues that may result from a migration. Refer to the Inspector Tool documentation for additional information on how to run the tool and view results.
Run the Formula Validation Tool
The Formula Validation tool identifies issues with the formula expression syntax in dashboards, business schemas, and schemas. In order to validate the formula expression syntax, the tool requires a tenant export file. The CMC Administrator can create the tenant export using the Cluster Management Console (CMC) or the System Administrator can create the tenant export using the Tenant Management Tool (TMT). For the purposes of migration, the System Administrator should run the Formula Validation tool before and after a migration. Refer to the Formula Validation Tool documentation for additional information on how to run the tool and view results.
Types of Migration
Migrate as a Developer
Developers typically migrate specific tenant objects, which is the simplest form of migration. Ideally use specified object migration in late phase development or in a continuous integration continuous deployment (CICD), or similar process. Also use it in early phase development if the DEV environment has a large number of extraneous objects not relevant to the UAT or PROD environments. The following are the tenant objects that can be independently moved from one environment to another, and the migration steps.
Migrate External Data Sources
Migrate external data sources with the Command Line Interface (CLI) tool. Use the export_datasources
command to export the source environment data source to a ZIP file, and the import_datasources
command to import the ZIP file to the target system. Refer to the CLI tool documentation for more information on accessing the tool and command usage.
Migrate Schemas
Migrate schemas with the Incorta Direct Data Platform™ user interface. Here are the schema migration steps in the Incorta Direct Data Platform™:
- In the Navigation bar of the source environment, select Schema.
- Select the checkbox to the left of the schema you would like to export.
- In the Action menu that appears (⋮ vertical ellipsis icon), select Export.
- In the Navigation bar of the target environment, select Schema.
- In the Action bar, select select + New → Import Schema.
- Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing schema checkbox if you are importing a schema with the same name as an existing schema.
- The Import Results dialog will contain a status of Created if the import was successful. Select Close.
You can also migrate schemas with the Command Line Interface (CLI) tool. Use the export_schemas
command to export the source environment schema to a ZIP file, and the import_schemas
command to import the ZIP file to the target environment. Refer to the CLI tool documentation for more information on accessing the tool and command usage.
Migrate schemas along with their files
You can migrate schemas along with their files on the shared storage. You must have root access to the host that runs the Analytics platform in both environments or ask the System Administer who has root access to copy the files.
Here are the steps to migrate schemas along with their files:
- Export the schema from the original environment using the platform's user interface or the CLI.
- Import the schema to the target environment using the platform's user interface or the CLI.Important
If the schema exists in both environments, do the following in the target environment before performing the next steps:
- Enable the Overwrite existing schema option when importing the schema.
- Delete the schema
source
directory from the shared storage. - Delete the schema records from the FILES_VERSIONS and VERSION_LOCK metadata database tables.
- Delete the schema's
time.log
file. The path of this file is/home/incorta/IncortaAnalytics/Tenants/<tenant_name>/<schema_name>
.
- Copy the schema's folder that exists under the tenant's
source
directory in the original environment to thesource
directory of the tenant in the target environment. - Copy the schema's
time.log
file from the original environment to the target environment. The path of this file must be/home/incorta/IncortaAnalytics/Tenants/<tenant_name>/<schema_name>
. - Load the schema from staging.
- When migrating shared storage files from one environment to another, you must load the schema from staging after copying the
source
directory. Copying theddm
andsource
directories will not have the same result. - To migrate only one object in a physical schema, you need to copy the whole object directory that exists under the physical schema in the
source
directory. The path to the object directory that you need to copy is as follows:/home/incorta/IncortaAnalytics/Tenants/<tenant_name>/source/<schema_name>/<object_name>
. - Both environments must run an Incorta release that supports file versioning.
- If the schema exists in both environments, you must replace the schema's
time.log
file in the target environment with the one in the original environment, and the copied files should not have records in the FILES_VERSIONS or VERSION_LOCK metadata database tables to ensure data consistency when loading the schema incrementally.
Migrate Business Schemas
Migrate business schemas with the Incorta Direct Data Platform™ user interface. Here are the business schema migration steps in the Incorta Direct Data Platform™:
- In the Navigation bar of the source environment, select Business Schema.
- Select the checkbox to the left of the business schema you would like to export.
- In the Action menu that appears (⋮ vertical ellipsis icon), select Export.
- In the Navigation bar of the target environment, select Business Schema.
- In the Action bar, select select + New → Import Business Schema.
- Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing business schema checkbox if you are importing a business schema with the same name as an existing business schema.
- The Import Results dialog will contain a status of Created if the import was successful. Select Close.
Migrate Schema Load Jobs
Migrate schema load jobs manually from one environment to another. You can recreate the schema load in the target environment based on the load job configuration in the source environment. Refer to the Scheduler documentation for additional information.
Migrate Schema Alerts
Migrate schema alerts manually from one environment to another. You can recreate the schema notification in the target environment based on the schema notification configuration in the source environment. Refer to the Scheduler documentation for additional information.
Migrate Dashboards
Migrate dashboards with the Incorta Direct Data Platform™ user interface. Here are the dashboard migration steps in the Incorta Direct Data Platform™:
- In the Navigation bar of the source environment, select Content.
- Select the dashboard you would like to migrate to open it.
- In the More Options menu (⋮ vertical ellipsis icon), select Export.
- In the Export Dashboard dialog, select Export.
- In the Navigation bar of the target environment, select Content.
- In the Action bar, select select + New → Import Folder/Dashboard.
- Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing dashboard checkbox if you are importing a dashboard with the same name as an existing dashboard.
- The Import Results dialog will contain a status of Created if the import was successful. Select Close.
You can also migrate dashboards with the Command Line Interface (CLI) tool. Use the export_dashboards
command to export the source environment dashboard to a ZIP file, and the import_dashboards
command to import the ZIP file to the target environment. Refer to the CLI tool documentation for more information on accessing the tool and command usage.
Migrate Dashboard Alerts
Migrate dashboard alerts with the Incorta Direct Data Platform™ user interface. Here are the Dashboard Alert migration steps in the Incorta Direct Data Platform™:
- In the Navigation bar of the source environment, select Scheduler, and then the Data Notifications tab.
- Select the checkbox to the left of the dashboard alert you would like to export.
- In the Action menu that appears, select Export (Download icon).
- In the Navigation bar of the target environment, select Scheduler, and then the Data Notifications tab.
- In the Action bar, select select + New → Import Notifications.
- Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing alerts checkbox if you are importing a data alert with the same name as an existing data alert.
- The Import Results dialog will contain a status of Created if the import was successful. Select Close.
Migrate Dashboard Delivery Jobs
Migrate dashboard delivery jobs with the Incorta Direct Data Platform™ user interface when you migrate the associated dashboard. Select the Include Scheduled Jobs checkbox in the Export dialog.
Migrate Bookmarks
Migrate bookmarks with the Incorta Direct Data Platform™ user interface when you migrate the associated dashboard. Select the Include Bookmarks checkbox in the Export dialog. Both public and private bookmarks will be preserved.
Migrate Global Variables
Migrate global variables manually from one environment to another. You can recreate the global variable in the target environment based on the global variable configuration in the source environment. Refer to the Schema Manager documentation for additional information.
Migrate Session Variables
Migrate internal and external session variables with the Command Line Interface (CLI) tool. The export_session_variables
command can be used to export the session variables to a ZIP file from the source environment, and the import_session_variables
command can be used to import the ZIP file to the target system. Refer to the CLI tool documentation for more information on accessing the tool and command usage.
Migrate Groups
Migrate groups manually from one environment to another. You can recreate the groups in the target environment based on the group configuration in the source environment. Refer to the User Profile Manager documentation for additional information.
Migrate Users
Migrate users manually from one environment to another. You can recreate the users in the target environment based on the user configuration in the source environment. Refer to the User Profile Manager documentation for additional information.
A more complex scenario involves the migration of more than one of the tenant objects listed above. It is critical to ensure that dependent objects are not broken when a change is made to a tenant object. For example, when you change a business schema column, it is important to migrate the affected dashboards. As mentioned above, the Inspector Tool will help to identify any migration issues.
Migrate as a Tenant Super User
Super Users typically migrate full tenants. Ideally use full tenant migration in early phase development for the initial creation of the UAT or PROD environments. The following are considerations when you migrate a full tenant:
- All objects are migrated with the original ownership. It is recommended that the objects in PROD are owned by Administrators or Super Users, not by developers.
- Data sources point to databases in DEV or UAT that are different than in PROD.
- There is not an option to exclude certain objects from migration.
There are two options for migrating a full tenant:
- Use the CMC
- Use the TMT
The CMC option is user interface driven, while the TMT is a command line utility. The other difference is the TMT has an option to copy local data files and folders as part of the migration, while the CMC does not. Before you migrate the full tenant with either option, it is important to first create a tenant backup.
Backup the Existing Tenant in the Target Environment
The following are the steps to create a tenant backup using the CMC:
- Sign in to the CMC of the source environment.
- In the Navigation bar, select Clusters.
- In the cluster list, select a Cluster name.
- In the canvas tabs, select Tenants.
- In the More Options menu (⋮ vertical ellipsis icon) at the end of the tenant row, select Backup.
Use the CMC to Migrate the Full Tenant
The following are the steps to migrate the full tenant using the CMC:
Sign in to the CMC of the source environment.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Tenants.
In the More Options menu (⋮ vertical ellipsis icon) on the right side of the tenant row, select Execute inspector now to review and verify the tenant object lineage.
In the More Options menu (⋮ vertical ellipsis icon) on the right side of the tenant row, select Export. The tenant will be packaged into a
tenant_<TENANT_NAME>.zip
folder containing:<TENANT_NAME>.properties
file<TENANT_NAME>.zip
file- dashboards folder containing XML
- datasources folder containing XML
- schemas folder containing XML
tenant.xml
file
The exported ZIP file will be downloaded to your browser’s default download folder.
Sign in to the CMC of the target environment.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Tenants.
Select + → Import Tenant.
Drag and drop the
tenant_<TENANT_NAME>.zip
to the Click or drag a file here to upload panel in the Import a tenant to the cluster dialog.Select Next.
If there is a
<TENANT_NAME>.properties
file located withintenant_<TENANT_NAME>.zip
, select Yes, Use it to autofill the tenant property values. The tenant property values can be changed later. Otherwise, select No, ignore it.Verify or enter the tenant properties:
Property Control Description Name text box Enter the tenant name. Select Check to determine if a Tenant already exists with the name entered. Username text box Enter the username for the Super User Password text box Enter the password for the Super User Email text box Enter the email address for the Super User Path text box Enter the shared storage path for tenant related data Pause scheduled jobs toggle Enable this property if the imported tenant will have all scheduled jobs paused on import Enter and verify the tenant email properties:
Property Control Description Sender’s Username Auth toggle Enable this property if the email requires username authentication System Email Username text box Enable Sender’s Username Auth to configure this property. Enter the username for the system email. System Email Address text box Enter the email address for the system email System Email Password text box Enter the password for the system email address SMTP Host text box Enter the Simple Mail Transfer Protocol (SMTP) host for the system email SMTP Port text box Enter the SMTP port number Share Notifications toggle Enable this property to share notifications Select Create to finalize the tenant migration.
ImportantYou may need to enter the username and password for some of your imported connectors that are in the tenant. This affects the following connectors:
- Amazon Web Services (AWS)
- RedShift
- Apache Drill
- Apache Hive
- Cassandra (Simba)
- Custom SQL
- IBM DB2
- IBM Netezza
- Microsoft SQL Server
- Microsoft SQL Server (JTDS)
- MongoDB BI
- MySQL
- NetSuite - SuiteAnalytics
- Oracle
- Presto
- PostgreSQL
- SAP Hana
- SAP Sybase IQ
- Teradata
- Vertica
If the tenant contained data files that were not part of the migration, they should be added to the indicated directory for tenant data:
<CMC_INSTALLATION_PATH>/IncortaAnalytics/Tenants/<Tenant_Name>/dataRun the Inspector Tool following the migration to identify any lineage errors that may have occurred from moving the tenant between environments.
Use the TMT to Migrate the Full Tenant
Another option to migrate a full tenant is to use the TMT command line tool. A System Administrator with root privileges to the host running the CMC is able to run the TMT.
TMT file path
Following is the default path to the TMT tool in your CMC:
<CMC_INSTALLATION_PATH>/IncortaAnalytics/cmc/tmt
TMT Command for Tenant Export
Here are the TMT commands to export the full tenant:
cd <CMC_INSTALLATION_PATH>/IncortaAnalytics/cmc/tmt/./tmt.sh --cluster-name <CLUSTER_NAME> --export <TENANT_EXPORT_NAME>.zip --cf
TMT Command for Tenant Import
Here is the TMT command to import the full tenant:
./tmt.sh -clnm <CLUSTER_NAME> -i <PATH_TO_TENANT_EXPORT_NAME>.zip
Refer to the TMT documentation for more information on command usage.
Migrate as a System Administrator
System Administrators typically migrate host files that contain configurations, properties, customizations, and more. This requires root access to the hosts. As necessary, copy host files manually from one environment to another, or edit the file manually in the target environment as indicated in the table below:
Host File | Migration Action |
---|---|
Linux Host Configurations - Ulimit | Manually edit/etc/security/limits.conf |
CMC Node Configuration files | Manually edit all files under<CMC_INSTALLATION_PATH>/cmc/cmcData Manually edit <CMC_INSTALLATION_PATH>/cmc/conf/catalina.properties |
Analytics and Loader Service Properties | Manually edit each of the following files:<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/ node.properties <INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/ services/<SERVICE_GUID>/conf/catalina.properties <INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/ services/<SERVICE_GUID>/incorta/service.properties <INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/ services/<SERVICE_GUID>/incorta/engine.properties <INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/ services/<SERVICE_GUID>/incorta/options.properties <INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/ nodeAgent/nodeAgent.cfg |
Spark Configuration | Manually edit<SPARK_NODE_INSTALLATION_PATH>/spark/conf/ spark-defaults.conf |
Zookeeper Configuration | Manually edit<ZOOKEEPER_NODE_INSTALLATION_PATH>/zookeeper/conf/ zoo.cfg |
Independent JARs | Copy the file to the appropriate connectors subdirectory<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/ extensions/connectors/ |
Tomcat server XML | Manually edit<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/ services/<SERVICE_GUID>/conf/server.xml |
CSS Customizations | Manually edit<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/ runtime/webapps/ROOT/css/custom-new.css |
Certificates, SSL PCKS12 or Java Keystore Files for an External Data Source | Copy the file to the certs directory/home/incorta/IncortaAnalytics/certs/ |
Migrate as a CMC Administrator
CMC Administrators typically migrate CMC configurations. CMC configurations are migrated manually from one environment to another. You can replicate the CMC configurations in the target environment based on the CMC configurations in the source environment. Adjust any configuration values as necessary for the target environment. The following are the CMC configurations to consider for migration.
An update of CMC configurations may require a restart of the analytics service, loader service, or all services.
Migrate General Configurations
Migrate Node Services Configurations
Migrate the On Heap Memory (GB), Off Heap Memory (GB), and CPU Utilization (%) for both the Analytic and Loader services as follows:
- Sign in to the CMC.
- In the Navigation bar, select Nodes.
- In the Services tab, select either an Analytics or Loader Service.
- Select Edit (Pen icon).
- Update the node services configurations as necessary.
- Select Update.
Migrate Notebook Add-on
Enable Notebook Add-on for for the Analytics service as follows:
- Sign in to the CMC.
- In the Navigation bar, select Nodes.
- Select the Add-ons tab.
- Select + → Add Notebook.
Migrate Server Cluster Configurations
Migrate the server cluster configurations as follows:
- Sign in to the CMC.
- In the Navigation bar, select Clusters.
- In the cluster list, select a Cluster name.
- In the canvas tabs, select Cluster Configurations.
- In the panel tabs, select Server Configurations.
- In the left pane, select the one of the following options to update the associated server configurations:
- Kafka Consumer Service Name
- SQL Interface
- Spark Integration
- Tuning
- UI Customizations
- Diagnostics
Migrate Default Tenant Cluster Configurations
Migrate the default tenant configurations as follows:
- Sign in to the CMC.
- In the Navigation bar, select Clusters.
- In the cluster list, select a Cluster name.
- In the canvas tabs, select Cluster Configurations.
- In the panel tabs, select Default Tenant Configurations.
- In the left pane, select one of the following options to update the associated default tenant configurations:
- Security - Authentication Type
- Regional Settings
- Email - SMTP Host, Enable SMTP SSL, Enable Notifications, Enable Internal Error Notifications
- Data Loading - Stop Loading on First Error, Pause Scheduled Jobs
- Integration - Google Maps API Key, Apple Maps Developer Team ID, Apple MapKit JS Key ID, Apple Maps API Key, Google Drive Client ID, Google Drive Client Secret, Dropbox App Key, Dropbox App Secret, Box Client ID, Box Client Secret, Mapbox API KeyTenant EFS
- Advanced - Insight Max Groups UI Default, Insight Max Groups Limit, Insight Max Values in Filter Subquery, Force Reload Columns, Sync In Background, Warmup Mode
- Tuning - Schema Pool Size, Table Maximum Parallel Chunks, Evaluate Session Variables At Login
- Incorta Labs - Enable Insight 'View As' Menu, Notebook Integration
- External Visualization - Incorta Host, Incorta Port