Tools → Data Manager

About the Data Manager

The Data Manager allows a tenant user with the necessary access permissions to search for, create, and manage external data sources, local data folders, local data files, data agents, and data destinations.

Note

The availability of the Data Agents, Data Destinations, and the related options is dependent on additional configurations in the Cluster Management Console (CMC).

Data Manager Access Permissions

A user that belongs to a group with the Schema Manager or the SuperRole role can access the Data Manager. To access the Data Manager for a given tenant, in the Navigation bar, select the Data tab.

Data Manager Anatomy

The Data Manager consists of the following:

  • Action bar
  • Context bar
  • Search bar
  • List view

Action bar

By default, the Action bar shows the context title, either External Data Sources or Data (for Local Data Files). For Local Data Files, the Action bar title is a navigable folder path.

+ New in the Action bar opens the Add New menu. The possible Add New menu options are:

  • Add Data Source
  • Add Data Destination
  • Create Folder
  • Add Data Agent

Context bar

The Context bar consists of three tabs: External Data Sources, Local Data Files, and Data Destinations. Tab selection determines the search context as well as the List view results.

The Search bar contains a search text box. Search for any part of a given search term. When the Context bar tab selection is External Data Sources, the search returns a result list of zero or more external data sources. When the tab selection is Local Data Files, the search returns a result list of zero or more local data folders and files. When the tab selection is Data Agents, the search returns a result list of zero or more data agents. When the tab selection is Data Destinations, the search returns a result list of zero or more data destinations.

The Search bar shows:

  • the number of items in the result list
  • the number of selected items in the result list with the option (x) to deselect the items
  • Share icon, if applicable, to open the Share dialog and assign or remove access right permission to the selected item. This icon is available only when you select one item.
  • Delete icon, to delete all selected items

List view

The List view shows all results for a given search term and search context. By default, the list view only show items where the given user has view access permissions to the items.

Note

Super User

Only the Super User tenant administrator can view all items, regardless of access permissions to the objects.

The List view shows 20 items at a time. For lists with more than 20 items, the view contains page controls. The page controls allow the user to navigate up to 5 pages (100 items) forwards or backwards as well as to the first page or to the last page of items.

External Data Sources list view

External data source items appear in alphabetical, ascending order. For External Data Sources, the List view shows the following properties:

PropertyControlDescription
Select AllcheckboxHeader control to add or remove all list view page items to a group for deletion or sharing
Select itemcheckboxSelect to add or remove and individual item to a group of zero or more items for deletion or sharing
Invalid Data SourceiconShows a warning when the external data source was saved with incomplete or invalid properties
Data Source ConnectoriconA logo icon for the data source connector
NametextThe name of the data source
TypetextThe name of the data source connector
OwnertextThe display name of the user creator
Test ConnectionbuttonTests the data source connection and returns a notification panel result regarding the success or failure of the test (a failure includes the exception details)
EditbuttonOpens the Edit Data Source dialog
SharebuttonOpens the Share dialog
DeletebuttonOpens the confirmation dialog for data source deletion

Local Data Files list view

By default, the Local data folder and file items appear in alphabetical, ascending order. To sort the items by the column values in either ascending or descending order, select a specific column header

For a given folder item, you can select the row, and navigate into the folder. The Action bar title shows a navigable folder path.

For Local Data Files, the list view shows the following properties:

PropertyControlDescription
Select AllcheckboxHeader control to add or remove all list view page items to a group for deletion or sharing
Select itemcheckboxSelect to add or remove and individual item to a group of zero or more items for deletion or sharing
Folder or FileiconIcon for folder or file
NametextThe name of the local data folder or file
TypetextThe file type (TXT, CSV, XLSX) or folder name of the local data folder or file
PermissiontextShow access right permissions: View, Share, Edit
OwnertextThe display name of the user creator
Last ModifiedtextDate and time or period ago (seconds, minutes, hours, and days)
SharebuttonOpens the Share dialog
DeletebuttonOpens the confirmation dialog for data source deletion

Data Agents list view

Data agent items appear in alphabetical, ascending order. For Data Agents, the List view shows the following properties:

PropertyControlDescription
Select AllcheckboxHeader control to add or remove all list view page items to a group for deletion
Select itemcheckboxSelect to add or remove an individual item to a group of zero or more items for deletion
NametextThe name of the data agent.
DescriptiontextThe description of the data destination
OwnertextThe display name of the user creator
StatusiconAn icon the indicated the status of connected or disconnected.
Generate Authentication FilebuttonSelect the button to generate or regenerate an authentication file for the data agent.
EditbuttonOpens the Edit Data Agent dialog
DeletebuttonOpens the confirmation dialog for data destination deletion

Data Destinations list view

Data destination items appear in alphabetical, ascending order. For Data Destinations, the List view shows the following properties:

PropertyControlDescription
Select AllcheckboxHeader control to add or remove all list view page items to a group for deletion
Select itemcheckboxSelect to add or remove an individual item to a group of zero or more items for deletion
File SystemiconA logo icon for the data destination type
NametextThe name of the data destination
TypetextThe type of the data destination
OwnertextThe display name of the user creator
Test ConnectionbuttonTests the data destination connection and returns a notification panel result regarding the success or failure of the test (a failure includes the exception details)
EditbuttonOpens the Edit Data Destination dialog
SharebuttonOpens the Share dialog
DeletebuttonOpens the confirmation dialog for data destination deletion

Data Manager Actions for external data sources

With the Data Manager, you can perform the following actions for External Data Sources:

Create an external data source

Before creating an external data source, review References → Connectors → for specific property definitions, configurations, and implementation steps.

Note

Validation Rules for a External Data Source name

An external data source name...

  • must be between 1 and 255 characters in length
  • must begin with an alpha character, lower or upper case
  • after the first alpha character, can contain zero or more alphanumeric characters in lower, upper, or mixed case
  • after the first alpha character, can contain zero or more underscore (_) characters
  • besides underscore (_), cannot contain special characters, symbols, and spaces

Here are the general steps to create an external data source:

  • In the Navigation bar, select Data.
  • In the Action bar, select + NewAdd Data Source.
  • In the Choose a Data Source dialog, select a data source connector from one the following categories:
    • Database
    • Application
    • File System
    • Query Service
    • Data Lake
    • Streaming Data
    • Other
    • Custom
  • In the New Data Source dialog, specify the properties for the given data source connector.
  • To test the connection, optionally select Test Connection.
  • To save, select Ok.
Note

Save anyway

You can create an External Data Source with invalid properties for the given data source connector. To save an external data source with invalid properties, select Save anyway.

View a list of external data sources

  • In the Navigation bar, select Data.
  • In the Context bar, select the External Data Source tab.
  • Review the list view of external data sources.

Search for an external data source

  • In the Navigation bar, select Data.
  • In the Context bar, select the External Data Source tab.
  • In the Search bar, enter a search term.
  • Review the list view of external data sources.

Test an external data source

  • In the list view of external data sources, select a given row.
  • Select Test Connection.

Edit an external data source

  • In the list view of external data sources, select a given row.
  • Select Edit.
  • In the Edit Data Source dialog, make the necessary configuration updates.
  • Optionally select Test Connection.
  • To save your changes, select Ok.

Share an external data source

  • In the list view of external data sources, select a given row.
  • Select Share.

In the Share dialog, you can:

  • remove an existing user or group.
  • modify an existing access permission for user or group.
  • add a new user or group with an access permission.

Delete an external data source

  • In the list view of external data sources, select a given row.
  • Select Delete (trash can).
  • In the dialog, select Delete.

Delete one or more external data sources

  • In the list view of external data sources, for each external data source that you want to delete, select the check box.
  • In the Search bar, next to Selected, select Delete (trash can).
  • In the dialog, select Delete.

Data Manager actions for local data folders and files

Using the Data Manager, you can perform the following actions for local data folders and files:

Create and upload one or more local data folders and or local data files, including subfolders and files

A folder can contain zero or more files with zero or more subfolders. Incorta preserves the hierarchy of folders. Incorta only uploads files with the following supported file extensions. After upload, Incorta will unzip compressed folders and files.

File ExtensionFile TypeExampleNotes
.csvcomma separated valuessales.csvCan contain a header row
.tsvtab separated valuessales.tsvCan contain a header row
.tabtab separated valuessales.tsvCan contain a header row
.txtcustom delimiter for separated valuessales.txtCan contain a header row
.xlsxMicrosoft Excel 2000 and abovesales.xlsxMust be .xlsx as .xls is not supported. Supports Worksheet selection.
.gzipcompressed file format for Linux and Unixsales.gzipArchive must contain a supported file type
.zipcompressed file format for Mac OS and Windowssales.zipArchive must contain a supported file type
Note

Header Row

A local data file can contain a header row. The Schema Wizard will use the first row as a header row. The Data Source dialog for a File System data source table has a header row property.

Here are the steps to create and one or more local data folders and/or local data files, including subfolders and files:

  • In the Navigation bar, select Data.
  • In the Action bar, select + NewAdd Data Source.
  • In the Choose a Data Source dialog, in Data Files, select Upload Data Folder.
  • In the Upload Data Folder dialog, in Upload Options, optionally select Overwrite existing file.
  • Drag and drop one or more parent folders or one or more files to the Upload Data Folder dialog.
Note

The Upload Data Folder option and dialog allow you to upload both data files and folders.

Create a local data folder

A local data folder can contain zero or more files with zero or more subfolders. To create a local data folder, follow these steps:

  • In the Navigation bar, select Data.
  • In the Action bar, select + NewCreate Folder.
  • In the Add Folder dialog, enter the local data folder Name.
  • Select Add to create the folder.
Note

Validation Rules for a Local Data Folder name

A local data folder name...

  • must be between 1 and 255 alphanumeric characters in length
  • can contain contain special characters, symbols, and spaces except for: /, ?, *, :, ", <, > and |.

View a list of local data folders and files

  • In the Navigation bar, select Data.
  • In the Context bar, select the Local Data Source tab.
  • Review the list view of local data folders and files.

Search for a local data folder or file

  • In the Navigation bar, select Data.
  • In the Context bar, select the Local Data Source tab.
  • In the Search bar, enter a search term.
  • In the Search result dialog, select a local data folder or file.
  • Review the list view of local data folders and files.
Note

Navigable folder path

When you select a folder from the Search result dialog, you navigate to that folder. The Action bar title shows a navigable folder path.

Share a local data folder or file

  • In the list view of local data folder or file, select a given row.
  • Select Share.

In the Share dialog, you can:

  • remove an existing user or group.
  • modify an existing access permission for user or group.
  • add a new user or group with an access permission.

Delete a local data folder or file

  • In the list view of local data folder or file, select a given row.
  • Select Delete (trash can).
  • In the dialog, select Delete.

Delete one or more local data folders or files

  • In the list view of local data file, for each local data file or folder that you want to delete, select the check box.
  • In the Search bar, next to Selected, select Delete (trash can).
  • In the dialog, select Delete.

Data Manager Actions For Data Agents

Starting with release 5.0, you can utilize a data agent to securely ingest data from an external data source that is behind a firewall.

You create a data agent in the Data Manager to authenticate and monitor a remote data agent service. Only a Tenant Administrator (Super User) or user that belongs to a group with the SuperRole role for a given tenant can create a data agent instance that connects to a data agent service. A user that belongs to a group with the Schema Manager role can see a list of data agents in the Data Manager.

Note

You can generate and regenerate an authentication file from the Data Manager for a given data agent. The file contains all the information needed by the data agent service to connect to the Incorta cluster. The file includes information regarding the various hosts, ports, and TLS/SSL certificates.

With the Data Manager, you can perform the following actions for Data Agents:

Create a data agent

Here are the steps to create a data agent in the Data Manager:

  • Sign in to the tenant as the Tenant Administrator (Super User) or user that belongs to a group with the SuperRole role.
  • In the Navigation bar, select Data.
  • In the Action bar, select + New > Add Data Agent.
  • In the Create Data Agent dialog, enter a Data Agent Name and optionally enter a description.
  • In the Generate Authentication File, select Generate Later or Generate Now.
Note

Validation Rules for a Data Agent name

A data agentname...

  • must begin with an alpha character
  • must be between 1 and 254 alphanumeric characters in length
  • can contain an _
  • cannot contain any special characters besides underscore, _.

View a list of data agents

  • In the Navigation bar, select Data.
  • In the Context bar, select the Data Agents tab.
  • Review the list view of data destinations.

Search for a data agent

  • In the Navigation bar, select Data.
  • In the Context bar, select the Data Agents tab.
  • In the Search bar, enter a search term.
  • Review the list view of data agents.

Generate an authentication file

You can generate or regenerate an authentication file from the Data Manager for a given data agent. The file contains all the information needed by the data agent service to connect to the Incorta cluster. The file includes information regarding the various hosts, ports, and TLS/SSL certificates.

  • In the Navigation bar, select Data.
  • In the Context bar, select the Data Agents tab.
  • In the list view of data agents, for a given data agent row, select Generate Authentication File or Regenerate Authentication File.
  • As required, move and copy the downloaded .auth file.

Edit a data agent

  • In the list view of data agents, select a given row.
  • Select Edit.
  • In the Edit Data Agent dialog, you can edit Description.
  • To save your changes, select Ok.

Delete a data agent

  • In the list view of data agents, select a given row.
  • Select Delete (trash can).
  • In the dialog, select Delete.

Delete one or more data agents

  • In the list view of data agents, for each data agent that you want to delete, select the check box.
  • In the Search bar, next to Selected, select Delete (trash can).
  • In the dialog, select Delete.

Data Manager Actions For Data Destinations

You can utilize data destinations to export supported insights to Microsoft OneDrive, FTP servers, SFTP servers, Google Drive, or Google Sheets. For more information, refer to Concepts → Data Destination

With the Data Manager, you can perform the following actions for Data Destinations:

Create a data destination

Note

Validation rules for a data destination name

A data destination name…

  • must be between 1 and 255 characters in length
  • must begin with an alpha character, lower or upper case
  • after the first alpha character, can contain zero or more alphanumeric characters in lower, upper, or mixed case
  • after the first alpha character, can contain zero or more underscore (_) characters
  • besides underscore (_), cannot contain special characters, symbols, and spaces

Here are the general steps to create a data destination:

  • In the Navigation bar, select Data.
  • In the Action bar, select + NewAdd Data Destination.
  • In the Choose a Data Destination dialog, select a data destination type from the following:
    • FTP
    • Google Drive
    • Google Sheets
    • OneDrive
    • SFTP
  • In the New Data Destination dialog, specify the properties for the given data destination type.
  • To test the connection, select Test Connection.
  • To save, select Ok.

For information about the available data destination properties per type, refer to Concepts → Data Destination → Data destination properties

View a list of data destinations

  • In the Navigation bar, select Data.
  • In the Context bar, select the Data Destinations tab.
  • Review the list view of data destinations.

Search for a data destination

  • In the Navigation bar, select Data.
  • In the Context bar, select the Data Destinations tab.
  • In the Search bar, enter a search term.
  • Review the list view of data destinations.

Test a data destination

  • In the list view of data destinations, select a given row.
  • Select Test Connection.

Edit a data destination

  • In the list view of data destinations, select a given row.
  • Select Edit.
  • In the Edit Data Destination dialog, make the necessary configuration updates.
  • Optionally, select Test Connection.
  • To save your changes, select Ok.

Share a data destination

  • In the list view of data destinations, select a given row.
  • Select Share.

In the Share dialog, you can:

  • remove an existing user or group.
  • modify an existing access right for a user or group.
  • add a new user or group with an access right.

Delete a data destination

  • In the list view of data destinations, select a given row.
  • Select Delete (trash can).
  • In the dialog, select Delete.

Delete one or more data destinations

  • In the list view of data destinations, for each data destination that you want to delete, select the check box.
  • In the Search bar, next to Selected, select Delete (trash can).
  • In the dialog, select Delete.