Connectors → Apache Hadoop

About Apache Hadoop

Apache Hadoop is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets. With the Apache Hadoop Web UI, you can typically access browse the HDFS file system.

Apache Hadoop Connector Updates

This section is to explore the updates in the newer versions of the Apache Hadoop connector available on the Incorta connectors marketplace.

In order to get the newer version of the connector, please update the connector using the marketplace.

Apache Hadoop Connector

The Apache Hadoop Connector enables Incorta to access files stored in an HDFS directory. Incorta is able to load the following file types from HDFS:

• Text (csv, tsv, tab, txt)
• Excel (xlsx)
• Parquet
• Optimized Row Columnar (ORC)

The Apache Hadoop connector supports the following Incorta-specific functionality:

Steps to connect Apache Hadoop and Incorta

To connect Apache Hadoop and Incorta, here are the high level steps, tools, and procedures:

• Create an external data source
• Create a schema with the Schema Wizard
• or, Create a schema with the Schema Designer
• Load the schema
• Explore the schema

Create an external data source

Here are the steps to create a external data source with the Apache Hadoop connector:

• Sign in to the Incorta Direct Data Platform.
• In the Navigation bar, select Data.
• In the Action bar, select + New → Add Data Source.
• In the Choose a Data Source dialog, in Data lake, select Data Lake - HDFS.
• In the New Data Source dialog, specify the applicable connector properties.
• To test, select Test Connection.
• Select Ok to save your changes.

Apache Hadoop connector properties

Here are the properties for the Apache Hadoop connector:

Create a schema with the Schema Wizard

Here are the steps to create an Apache Hadoop schema with the Schema Wizard:

• Sign in to the Incorta Direct Data Platform.
• In the Navigation bar, select Schema.
• In the Action bar, select + New → Schema Wizard
• In (1) Choose a Source, specify the following:
  • For Enter a name, enter the schema name.
  • For Select a Datasource, select the Apache Hadoop external data source.
  • Optionally create a description.
• In the Schema Wizard footer, select Next.
• In (2) Manage Tables, in the Data Panel, navigate the directory tree as necessary to select the Apache Hadoop files. You can either check the Select All checkbox or select individual sheets.
• In the Schema Wizard footer, select Next.
• In (3) Finalize, in the Schema Wizard footer, select Create Schema.

Create a schema with the Schema Designer

Here are the steps to create an Apache Hadoop schema using the Schema Designer:

• Sign in to the Incorta Direct Data Platform.
• In the Navigation bar, select Schema.
• In the Action bar, select + New → Create Schema.
• In Name, specify the schema name, and select Save.
• In Start adding tables to your schema, select Data Lake.
• In the Data Source dialog, specify the Apache Hadoop table data source properties.
• Select Add.
• In the Table Editor, in the Table Summary section, enter the table name.
• To save your changes, select Done in the Action bar.

Apache Hadoop table data source properties

For a schema table in Incorta, you can define the following Apache Hadoop specific data source properties as follows:

Summary of Data Access Methods Based on Remote and Performance Optimized Settings

Incremental Extract Methods

• Last Successful Extract Time: This option will load data from the time the last successful extract occurred.

  Here is an example use case of Last Successful Extract Time:

  • A directory containing all the sales data is located at /path/to/sales
  • The directory contains the following files: /path/to/sales/sales_california.parquet, /path/to/sales/sales_newyork.parquet, /path/to/sales/illinois.parquet

  When you perform a full load, the union of all existing files will be extracted into the same table. After that, if the directory receives a new file, such as /path/to/sales/sales_ohio.parquet, the next incremental load will pick up this file since its last modified timestamp will be more recent than that of the files extracted in the previous full load.

• Timestamp in File Name: This option will load data from the time specified in the file name.

  Here is an example use case of Timestamp in File Name:

  • A directory containing all the sales data is located at /path/to/sales
  • The directory receives a new file on daily basis: /path/to/sales/sales_2020-04-01.parquet, /path/to/sales/sales_2020-04-02.parquet, /path/to/sales/sales_2020-04-03.parquet

  When you perform a full load, the union of all existing files will be extracted into the same table. After that, if the directory receives a new file, such as /path/to/sales/sales_2020-04-04.parquet, the next incremental load will pick up this file since the timestamp in the file name is more recent than that of the files extracted in the previous full load.

Timestamp Formats in File Name

• yyyy-MM-dd
• dd.MM.yyyy
• dd-MMM-yy
• dd-MMM-yyyy
• yyyy-MM-dd HH.mm.ss
• Unix Epoch (seconds)
• Unix Epoch (milliseconds)

Text File Date Format

• yyyy-MM-dd
• dd/MM/yyyy
• dd.MM.yyyy
• dd/MMM/yyyy
• dd-MMM-yy
• dd-MMM-yyyy
• MM/dd/yyyy
• yyyy/MM/dd
• Unix Epoch (seconds
• Unix Epoch (milliseconds)
• Other

Text File Timestamp Format

• yyyy-MM-dd HH:mm:ss
• yyyy-MM-dd HH.mm.ss
• yyyy-MM-dd HH:mm:ss.SSS
• dd/MM/yyyy HH:mm:ss
• dd/MM/yyyy HH.mm.ss
• dd/MM/yyyy HH:mm:ss.SSS
• Unix Epoch (seconds
• Unix Epoch (milliseconds)
• Other

Text File Character Set

• US-ASCII
• ISO-8859-1
• UTF-8
• UTF-16BE
• UTF-16LE
• UTF-16

Text File Separator

• Comma
• Tab
• Other

View the schema diagram with the Schema Diagram Viewer

Here are the steps to view the schema diagram using the Schema Diagram Viewer:

• Sign in to the Incorta Direct Data Platform.
• In the Navigation bar, select Schema.
• In the list of schemas, select the Apache Hadoop schema.
• In the Schema Designer, in the Action bar, select Diagram.

Load the schema

Here are the steps to perform a Full Load of the Apache Hadoop schema using the Schema Designer:

• Sign in to the Incorta Direct Data Platform.
• In the Navigation bar, select Schema.
• In the list of schemas, select the Apache Hadoop schema.
• In the Schema Designer, in the Action bar, select Load → Full Load.
• To review the load status, in Last Load Status, select the date.

Explore the schema

With the full load of the Apache Hadoop schema complete, you can use the Analyzer to explore the schema, create your first insight, and save the insight to a new dashboard.

To open the Analyzer from the schema, follow these steps:

• In the Navigation bar, select Schema.
• In the Schema Manager, in the List view, select the Apache Hadoop schema.
• In the Schema Designer, in the Action bar, select Explore Data.