References → Incorta Connector Software Development Kit

The Incorta Connector Software Development Kit, or Connector SDK, allows Incorta customers and partners to develop their own custom connectors. 

With the Connector SDK, you can implement metadata discovery and data extraction that enables you to:

• Discover source database tables using the schema wizard
• Discover table columns while manually editing a schema table
• Execute full and incremental load queries on a source system
• Fetch data records to store in Incorta’s persistence layer

Follow the instructions in this article to learn how to implement, build, and deploy the connector.

Connector SDK Versions

Ensure you use the correct Connector SDK version with the correct release version of Incorta.

Implement the Connector

1. Download the Connector SDK JAR file. The JAR file, incorta.connector.api.jar, is offered independently.   If you need assistance, contact the Cloud Support.
2. Implement the following interfaces:
   • com.incorta.connector.ConnectorFactory
     • Main entry point for the connector. It is the factory class used for accessing the connector model and for instantiating the connector.
     • A single factory class can support multiple connector types.
   • com.incorta.connector.Connector
     • Main connector object that manages connection to the data source.
     • Use this parameter to discover source tables using the schema wizard and to create data set objects.
   • com.incorta.connector.DataSet
     • Extracts data for a single physical schema table.
     • It contains the necessary information for extracting the table data. For example, for an SQL-based connector, a data set definition contains the SQL query for that table.
     • A data set can support incremental extraction.
   • com.incorta.util.Record.RecordSet
     • A record set object represents the set of records returned from a single execution of a data set. Create a record set during the ETL job. The record set is consumed by Incorta’s loader. The record set provides an interface for the loader to iterate over the returned records.

Build the Connector

Package the connector into a JAR file. The connector JAR must have a manifest file that:

• Points to the connector factory class
• Specifies the connector API version

If the connector requires third-party libraries, you do not need to package them inside the connector JAR. Only deploy these libraries with the connector JAR.

You do not need to deploy the Connector API JAR with the connector and its dependencies, because it is included in your Incorta installation.

Below is a sample connector JAR manifest file. It specifies the connector factory class named SampleConnectorFactory, and indicates that it targets API version 1.0:

Copy
Manifest-Version: 1.0
Connector-Factory-Class: com.incorta.connector.sample.SampleConnectorFactory
Connector-API-Version: 1.0

Deploy the Connector

Deploy the connector on each Incorta node and the Cluster Management Console (CMC). The CMCs need a connector to access its model when you import or export tenants.

1. Copy the connector JAR, containing the connector implementation, manifest file, and any third-party library JARs into the following directories:

   Copy
   */IncortaNode/extensions/connectors/<connector_directory>/<connector.jar file>
   
   */cmc/extensions/connectors/<connector_directory>/<connector.jar file>
   
   

   Optionally, you can create a subdirectory in the connectors directory and copy the files there.

2. Restart Incorta services.

3. After restarting your services you can add the connector as a data source using the instructions in Connect to a Data Source.

The connector framework scans all directories recursively. A special class loader is used to load classes from the directories to mirror the directory structure. JARs located in separate directories are loaded using separate class loaders. JARs in a subdirectory are loaded using a child class loader.

This is done to ensure isolation between connectors. For example, if one connector uses one version of Apache Commons, while another connector uses a different version of Apache Commons, we want to make sure that each connector code accesses its own correct version.

Connector Model XML Structure

Configuring the connector is done with an XML file.

It is structured as follows: Root element is a <connectorModel> tag, containing one <connectorProperties> tag and one <datasetProperties> tag.

Each one of them contains zero or more <propertyModel> tags that describe the properties of the connector or data set.

Copy
<?xml version="1.0" ?>
<connectorModel version="1" typeName="sql" typeLabel="SQL Database" subTypeName="mysql" subTypeLabel="MySQL">
   <connectorProperties>
       <propertyModel type="TEXT" name="user" label="Username" required="true" />
       <propertyModel ... />
       ...
   </connectorProperties>
   <dataSetProperties>
       <propertyModel type="LONG_TEXT" name="query" label="Query" required="true" />
       <propertyModel ... />
       ...
   </dataSetProperties>
</connectorModel>

Tags and Attributes

Tag: connectorModel

This is the root tag of the connector model.

<connectorModel> tag attributes

displaySourceFormat

This is the format of the text displayed under the data set icon that appears when you edit a schema table.

Usually it is just the connector name, that is why the default value is ${connector}, which is just a placeholder for the connector name.

In some cases, however, it is desired to display more details, such as in the file system connector where the file name is also displayed.

Placeholders in the format string are substituted with the connector name (in case of ${connector} placeholder) or data set property names (e.g. ${filename}).

predefined

This flag indicates that the connector instance is created and maintained by Incorta, not by the user. There can be a single connector instance of this type per tenant, and it is named after the subTypeLabel value.

This field is used mainly by local file system connector, because local files are stored in one place that can't be changed.

Tag: connectorProperties

A required container tag for connector properties (propertyModel tags). The order of the properties inside this tag is the same as the order of the property fields as displayed in the UI. This tag has no attributes.

Tag: datasetProperties

A required container tag for data set properties (propertyModel tags). The order of the properties inside this tag is the same as the order of the property fields as displayed in the UI. This tag has no attributes.

Tag: propertyModel

This tag is used to define a single property field that must be displayed in the UI.

Table of supported types for the <propertyModel> tag attributes

In case a property is of type LIST or LIST_WITH_OTHER, the property must also have a child possibleValues tag to list the property values.

A property can also be dependent on other properties. It can be displayed by the UI only if specific values were selected in other properties. This is done by specifying those dependencies in the child dependencies tag.

Property Attributes

Table of <propertyModel> attributes

Tag: possibleValues

A container tag for predefined property values (propertyValue tags). This tag is used only for properties of type LIST or LIST_WITH_OTHER.

Tag: propertyValue

Represents a single predefined value for a property of type LIST or LIST_WITH_OTHER.

Tag: dependencies

An optional container tag for property dependencies (propertyDependency tags). It is used when we need to make a field appear only when user selects a specific value for another property.

For example, the queryUpdate property that contains the incremental query should be displayed only if the incremental property is set to true.

There can be more than one dependency. The current property will be displayed only if all dependencies are satisfied.

Tag: propertyDependency

Represents a single property dependency.