Tools → Data Studio Manager

About the Data Studio Manager

The Data Studio Manager is an Incorta Premium Feature that allows users to easily create Materialized Views (MVs) for their data using a simple drag-and-drop interface.

With Data Studio, users can perform various transformations on their data without complex coding, by connecting a series of analytical steps called Recipes. Starting with the base table loaded into Incorta, the data undergoes a series of transformations in which each step applies specific recipes to refine, shape, and analyze the data. This streamlined progression ensures that data seamlessly transitions from its initial state to a refined, actionable form within Data Studio.

Data Studio Manager Access Permissions

Only an admin can enable Data Studio within the Cluster Management Console (CMC). For more information, refer to Enabling the Data Studio.

→ To access the Data Studio Manager, in the Navigation bar, select Data Studio. This opens the landing page where users manage multiple data flows.

Data Studio Manager Anatomy

The Data Studio Manager consists of the following:

• Action bar
• Search bar
• List view

Action bar

Select + New in the Action bar to open the Add New menu. The available options are:

• Create Data Flow
• Import Data Flow

Search bar

The Search bar contains a search text box for finding existing data flows by name. The results include data flows you own or have view, share, or edit access to. You can scroll through the list to find your desired data flow.

List view

Each data flow appears as a row in the list view. The following table shows the available properties:

The default sort in the list view is ascending by Name.
To change the sort for data flows, select the Name, Owner, Created, or Last Modified column heading, and change the arrow to point down to sort descending, and up to sort ascending.

Data Studio Manager Actions

Using the Data Studio Manager, you can perform the following actions:

• Create data flows
• Import data flows
• Edit data flows
• Connect or disconnect data flows
• Export data flows
• Delete data flows

Create data flows

1. In the Navigation bar, select Data Studio.
2. In the Action bar, select + New → Create Data Flow.
3. In the Create Data Flow dialog:
   • In the Name field, enter a unique name of the data flow (note: names are case-sensitive).
   • In the Description field, optionally provide a brief description.
4. Select Create.
5. The Data Flow Editor opens, allowing you to begin building your flow.

Import data flows

1. In the Navigation bar, select Data Studio.
2. In the Action bar, select + New → Import Data Flow.
3. Under Import Options, check the Overwrite existing dataflows box if you want to replace an existing data flow with the same name.
4. In the Import Data Flow dialog, perform one of the following:
   • Drag <dataflow_name>.zip to the Import Data Flow dialog, or
   • Click inside the Click or drag a file here to upload area, then browse to the location of the <dataflow_name>.zip, and select Open.
5. Wait for the upload to complete.
6. In the Import Results dialog, review the operation status.
7. Select Import.
   A confirmation message appears: Dataflow(s) imported successfully!

Edit data flows

1. Select More Options (⋮) → Edit for the desired data flow.
2. In the Edit Dataflow dialog, modify the Name and Description fields.
3. Select Edit.
   A confirmation message appears: Dataflow(s) edited successfully!

Connect or disconnect data flows

• To connect a data flow: Select the desired data flow to initiate and connect it.
• To disconnect a data flow: Select More Options (⋮) → Disconnect for the desired data flow.

Export data flows

Export multiple data flows

1. Select the checkboxes next to the desired data flows.
2. In the Action Bar, select More Options (⋮) → Export to download them as a .zip file.

Export a single data flow

For a specific data flow, select More Options (⋮) → Export to download it as a .zip file.

Delete data flows

Delete multiple data flows

1. Select the checkboxes next to the desired data flows.
2. In the Action Bar, select More Options (⋮) → Delete.
3. A confirmation prompt appears. Select Yes to permanently delete the data flows.

Delete a single data flow

1. For a specific flow, select More Options (⋮) → Delete.
2. A confirmation prompt appears. Select Yes to permanently delete the data flow.

Data Flow Editor Anatomy

To access the Data Flow Editor, open a specific data flow from the Data Studio Manager.
The interface consists of the following key components:

• Action bar
• View toolbar
• Canvas
• Overview panel
• Recipe panel
  • Results pane
• Data panel

Action bar

The Action bar provides key actions for managing your data flow. It includes the following controls:

View toolbar

The View toolbar provides navigation and layout controls for interacting with the data flow within the Canvas:

Canvas

The Canvas serves as the central area where you build your data flow and connect data recipes. It displays all data flow components, including recipes and joins. Add recipes by selecting + Recipe in the Action bar or by dragging and dropping datasets from the Data panel.

Overview panel

The Overview panel displays a zoomed-out view of your entire data flow. Select any area within the Overview panel to instantly zoom into that section on the Canvas.
The Overview panel is located in the upper-left corner of the Canvas. Select the arrow icon to expand or collapse the panel as needed.

Recipe panel

The Recipe panel appears on the right side of the Canvas when you select a recipe. It displays contextual actions and information specific to the selected recipe. The Recipe panel includes the following features:

Results pane

The Results pane provides a detailed view of the recipe’s output. It opens at the bottom of the Canvas when you select Explore in the Recipe panel, and includes:

• Result Set — Displays output records in a paginated table
• Profiling View — Offers insights into the result set through statistics, histograms, frequencies, and patterns depending on the dataset.
• Filter — Enables filtering data by selecting specific columns.

Data panel

The Data panel manages datasets and allows dragging them onto the Canvas to create new recipes. It is located on the left side of the View Toolbar.

Data Flow Editor Actions

Using Data Flow Editor, you can perform the following actions to build and manage your data flow:

• Add a new recipe
• Add a dataset as a recipe
• Create a Join recipe
• Filter data and create a shortcut recipe
• Configure data flow settings
• Preview code for a recipe
• Re-validate data flow
• Re-validate a recipe
• Deploy all MVs
• Refresh schemas

Add a new Recipe

1. In the Action bar, select + Recipe.
2. Choose a recipe type from the categories below, then configure its setting:

Note: The recipe name links to its configuration guide.

3. Select Save.
A confirmation message appears: Recipe added successfully!

Add a dataset as a recipe

1. Select the Data Panel icon next to the View Toolbar to expand the Data Panel.
2. In the Manage Dataset panel, select the checkboxes next to the tables you want to add.
3. Add a recipe to the Canvas using one of the following methods:
   • Drag and drop a dataset from the Data Panel onto the canvas, or
   • Select the (+) icon next to a dataset in the Data Panel.

This creates a recipe on the canvas based on the selected dataset.

Create a Join recipe

1. Select and drag from one recipe to another.
   This action automatically creates a Join recipe between the two datasets.
2. Configure the Join recipe settings, including:
   • Recipe Name
   • Join Type
   • Left and Right Input
   • Match On
   • Join Condition

For more information on configuring the Join Recipe, refer to References → Join Recipe.

Filter data and create shortcut recipes

1. Select a recipe on the Canvas.
2. In the Recipe panel, select Explore.
3. In the Results pane, select the Filter icon.
4. Select the column(s) you want to filter.
5. From the Selection Type dropdown menu, choose:
   • Include — keeps only selected values
   • Exclude — removes selected values
6. Select Apply. The Filter data appears in the Result Set pane.
7. Select Save ( 💾) to create a new recipe based on this filter.
8. Enter a Name for the new recipe shortcut.
9. Select Save.
   A confirmation message appears: Recipe added successfully!

Preview code for a recipe

1. Select a recipe on the Canvas.
2. In the Recipe panel, select the Preview Code icon.
3. View the auto-generated script that represents the transformation logic of the selected recipe.
4. Select X to close the code view.

Configure data flow settings

1. In the Action bar, select Settings.

2. In the Settings dialog:

   • Enable Sampling is toggled on by default with a sample size of 1000.
   • Edit the Sample Size based on your performance and profiling needs.
     Note

     Disabling sampling or increasing the sample size may slow down execution.

3. Edit the Sample Size as needed.

4. Select Save or Save & Restart to apply the changes.
   A confirmation message appears: Sampling size was changed successfully. Results will be updated after re-initializing the dataflow.

Re-validate data flow

1. In the Action bar, select More Options (⋮) → Re-validate Dataflow.
2. A confirmation prompt appears. Select Yes to revalidate your entire data flow.

Re-validate a recipe

5. Select a recipe on the Canvas
6. In the Recipe panel, select the Re-validate icon.
7. A confirmation prompt appears. Select Yes to revalidate your selected recipe.

Deploy all MVs

1. In the Action bar, select More Options (⋮) → Deploy All MVs.
2. A confirmation prompt appears. Select Yes to deploy all MV recipes in the data flow.

Refresh schemas

1. In the Action bar, select More Options (⋮) → Refresh Schemas.
2. A confirmation prompt appears. Select Yes to refresh the schemas.

Best Practices

• Materialized View (MV) deployment strategies
  Choose between:
  • Updates via data flow.
  • Static data flow, where updates are managed by editing in the Notebook.
• Data sampling
  • By default, data sampling is limited to 1,000 records for profiling to ensure optimal performance.
  • Disabling sampling can impact performance.
• Naming conventions
  • Materialized Views (MVs) can be traced back to their data flows.
  • Use descriptive and consistent names for data flows and MVs.