Tools → Upload Service Command Line Interface (CLI)

About the Incorta Upload Service CLI

The Incorta Upload Service CLI uploads data files from an Incorta On-Premises host or remote server to an Incorta Cloud Google Cloud Storage (GCS) bucket. You are able to access the Incorta Cloud GCS bucket from your Incorta Cloud instance. Following are the use cases for the Upload Service CLI:

You are an on-premises customer who would like to migrate from Incorta On-Premises to Incorta Cloud.
You are an on-premises customer who would like to sync data between Incorta On-Premises and Incorta Cloud for disaster recovery purposes.
You are a new customer who would like to upload data from a remote server to Incorta Cloud.

Install the Upload Service CLI

Prerequisites

In order to install the Upload Service CLI you must have the following:

Python3
A Python package manager, such as pip

Here are the steps to install the Upload Service CLI:

Download the CLI Python Wheel package on your machine.
Navigate to the Wheel package and install it using pip:
```
pip3 install ./incorta_cloud-<VERSION_NUMBER>-py3-none-any.whl
```
Replace <VERSION_NUMBER> with the version of the Wheel package you are installing.
Add the Python Binaries to your path:
- Get your Python installation path:
```
which python3
```
  As an example, the Python installation path returned is /Users/foo/Library/Python/3.7/bin/python3
- Set the path:
```
export PATH="/Users/foo/Library/Python/3.7/bin:$PATH"
```
  Note
  You can add the export line to your .bashrc or .zshrc so that you do not need to execute it every time you open a new shell session.

Configure the Upload Service CLI

Before you use the CLI, perform the following steps:

Set the environment variable
Sign in to the Upload Service CLI
Set the context

Set the Environment Variable

Set the INCORTA_CLOUD_CONFIG environment variable. This variable should hold the path of the file that the CLI will use to store the authentication configuration.

export INCORTA_CLOUD_CONFIG=<CONFIGURATION_FILE_PATH>

Here is an example of how to set the environment variable with a configuration file path of /Users/foo/incorta_config.json:

export INCORTA\_CLOUD\_CONFIG=/Users/foo/incorta_config.json

If the configuration file does not exist before you set the environment variable, it will be created. However, the path to the file must exist.

Note

You can add the export line to your .bashrc or .zshrc so that you do not need to execute it every time you open a new shell session.

incorta-cloud auth signin

You will be prompted to enter your username and password, which are the same credentials you use to access the Incorta Cloud Admin Portal.

Note

The credentials will not be stored on your machine. Instead, the auth token received from the server will be stored in the incorta_config.json file.

Set the Context

The context consists of the name of the instance and the tenant you intend to upload your data to. This information is stored in the configuration file so that you do not need to include it when you run the upload or sync.

Important

Both the instance and the tenant must exist before you set the context. If you are an on-premises customer migrating a tenant, refer to Additional Information for creating a tenant through tenant export and import.

Set the context as follows:

incorta-cloud context set-context --instance-name <instance_name> --tenant-name <tenant_name>

Upload Tenant Data

Here is the command to upload your tenant data:

incorta-cloud storage upload-tenant --root-path <ROOT_TENANT_DIRECTORY_PATH>

The CLI will list all of the files in the directory and will upload each to your GCS bucket. The data will be uploaded to the instance and tenant for which you set the context.

Upload a Data File

Here is the command to upload a single data file:

incorta-cloud storage upload --root-path <ROOT_FILE_DIRECTORY_PATH>

Sync Tenant Data

When you sync tenant data, your local directory and GCS bucket will both have the same data, with changes in one being reflected in the other. Here is the command to sync tenant data:

incorta-cloud storage sync-tenant-data --root-path <ROOT_TENANT_DIRECTORY_PATH> --conflict-resolution=<CONFLICT_RESOLUTION_PARAMETER>

There are three possible parameters for conflict resolution:

prefer-cloud: The cloud data file will be downloaded to the local directory and the local version will be overwritten.
prefer-local: The local data file will be uploaded to the cloud and the cloud version will be overwritten.
prefer-latest: The file that has the latest modification date will be kept and the other will be overwritten.

As an example, consider the following scenario:

Local Data Files	Cloud Data Files
file1.csv	file1.csv
file2.csv	file3.csv

When you run the sync command, file2.csv will be uploaded to the cloud, and file3.csv will be downloaded to the local directory.

For file1.csv, there are two possible cases:

The file contents are the same, in which no change occurs
The file contents are different, in which a conflict arises. The conflict resolution option specified in the sync command determines which file will be kept and which will be overwritten.

Sync Tenant Parquet

Here is the command to sync your tenant parquet:

incorta-cloud storage sync-tenant-data --root-path <ROOT_TENANT_DIRECTORY_PATH> --conflict-resolution=<CONFLICT_RESOLUTION_OPTION> --<PARQUET_DIRECTORY_PARAMETER>

If you want to sync your Parquet folder, replace <PARQUET_DIRECTORY_PARAMETER> with parquet-dir. If you want to sync your Compacted folder, replace <PARQUET_DIRECTORY_PARAMETER> with compacted-dir.

Review your Data

After you run a data upload or sync, you can go to your Incorta Cloud instance and sign in to your tenant. The data files will be available in Data → Local Data Files.

-->

Additional Information

If you are an on-premises customer migrating a tenant, you can create a tenant by exporting a tenant from Incorta On-Premises, and then importing the tenant to Incorta Cloud.

Export a tenant from Incorta On-Premises

Sign in to the CMC of the on-premises environment.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Tenants.
In the More Options (⋮ vertical ellipsis icon) menu on the right side of the tenant row, select Execute inspector now to review and verify the tenant object lineage.
In the More Options (⋮ vertical ellipsis icon) menu on the right side of the tenant row, select Export. The tenant will be packaged into a tenant_<TENANT_NAME>.zip folder containing:
- <TENANT_NAME>.properties file
- <TENANT_NAME>.zip file
  - dashboards folder containing XML
  - datasources folder containing XML
  - schemas folder containing XML
  - tenant.xml file
The exported ZIP file will be downloaded to your browser’s default download folder.

Import a tenant to Incorta Cloud

Sign in to the CMC of the Incorta Cloud environment.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
In the canvas tabs, select Tenants.
Select + → Import Tenant.
Drag and drop the tenant_<TENANT_NAME>.zip to the Click or drag a file here to upload panel in the Import a tenant to the cluster dialog.
Select Next.
If there is a <TENANT_NAME>.properties file located within tenant_<TENANT_NAME>.zip, select Yes, Use it. You can change the tenant property values later. Otherwise, select No, ignore it.
Verify or enter the tenant and tenant email properties.
Select Create to finalize the tenant import.