References → Incorta Connector Software Development Kit

The Incorta Connector Software Development Kit, or Connector SDK, allows Incorta customers and partners to develop their own custom connectors. 

With the Connector SDK, you can implement metadata discovery and data extraction that enables you to:

  • Discover source database tables using the schema wizard
  • Discover table columns while manually editing a schema table
  • Execute full and incremental load queries on a source system
  • Fetch data records to store in Incorta’s persistence layer

Follow the instructions in this article to learn how to implement, build, and deploy the connector.

Connector SDK Versions

Ensure you use the correct Connector SDK version with the correct release version of Incorta.

Implement the Connector

  1. Download the Connector SDK JAR file. The JAR file, incorta.connector.api.jar, is offered independently.   If you need assistance, contact the Cloud Support.
  2. Implement the following interfaces:
    • com.incorta.connector.ConnectorFactory
      • Main entry point for the connector. It is the factory class used for accessing the connector model and for instantiating the connector.
      • A single factory class can support multiple connector types.
    • com.incorta.connector.Connector
      • Main connector object that manages connection to the data source.
      • Use this parameter to discover source tables using the schema wizard and to create data set objects.
    • com.incorta.connector.DataSet
      • Extracts data for a single physical schema table.
      • It contains the necessary information for extracting the table data. For example, for an SQL-based connector, a data set definition contains the SQL query for that table.
      • A data set can support incremental extraction.
    • com.incorta.util.Record.RecordSet
      • A record set object represents the set of records returned from a single execution of a data set. Create a record set during the ETL job. The record set is consumed by Incorta’s loader. The record set provides an interface for the loader to iterate over the returned records.

Build the Connector

Package the connector into a JAR file. The connector JAR must have a manifest file that:

  • Points to the connector factory class
  • Specifies the connector API version

If the connector requires third-party libraries, you do not need to package them inside the connector JAR. Only deploy these libraries with the connector JAR.

You do not need to deploy the Connector API JAR with the connector and its dependencies, because it is included in your Incorta installation.

Below is a sample connector JAR manifest file. It specifies the connector factory class named SampleConnectorFactory, and indicates that it targets API version 1.0:

Manifest-Version: 1.0
Connector-Factory-Class: com.incorta.connector.sample.SampleConnectorFactory
Connector-API-Version: 1.0

Deploy the Connector

Deploy the connector on each Incorta node and the Cluster Management Console (CMC). The CMCs need a connector to access its model when you import or export tenants.

  1. Copy the connector JAR, containing the connector implementation, manifest file, and any third-party library JARs into the following directories:

    */IncortaNode/extensions/connectors/<connector_directory>/<connector.jar file>
    */cmc/extensions/connectors/<connector_directory>/<connector.jar file>

    Optionally, you can create a subdirectory in the connectors directory and copy the files there.

  2. Restart Incorta services.

  3. After restarting your services you can add the connector as a data source using the instructions in Connect to a Data Source.

The connector framework scans all directories recursively. A special class loader is used to load classes from the directories to mirror the directory structure. JARs located in separate directories are loaded using separate class loaders. JARs in a subdirectory are loaded using a child class loader.

This is done to ensure isolation between connectors. For example, if one connector uses one version of Apache Commons, while another connector uses a different version of Apache Commons, we want to make sure that each connector code accesses its own correct version.

Connector Model XML Structure

Configuring the connector is done with an XML file.

It is structured as follows: Root element is a <connectorModel> tag, containing one <connectorProperties> tag and one <datasetProperties> tag.

Each one of them contains zero or more <propertyModel> tags that describe the properties of the connector or data set.

<?xml version="1.0" ?>
<connectorModel version="1" typeName="sql" typeLabel="SQL Database" subTypeName="mysql" subTypeLabel="MySQL">
<connectorProperties>
<propertyModel type="TEXT" name="user" label="Username" required="true" />
<propertyModel ... />
...
</connectorProperties>
<dataSetProperties>
<propertyModel type="LONG_TEXT" name="query" label="Query" required="true" />
<propertyModel ... />
...
</dataSetProperties>
</connectorModel>

Tags and Attributes

Tag: connectorModel

This is the root tag of the connector model.

<connectorModel> tag attributes

AttributeRequiredDefault ValueDescription
versionNo1The connector version number (not the API version)
typeNameYesConnector type name (e.g. "sql")
typeLabelYesConnector type label displayed in UI (e.g. "SQL")
subTypeNameYesConnector sub-type name (e.g. "mysql")
subTypeLabelYesConnector sub-type label displayed in UI (e.g. "MySQL")
displaySourceFormatNo${connector}Format text under datasets icon. See displaySourceFormat below.
predefinedNofalseFlag indicating connector ownership. See predefined below.
activeNofalseIf set to false (default), the connector object will not be created until it is needed. When set to true, the connector object is initialized as soon as it is defined, or right after server startup, and stays available in memory.

displaySourceFormat

This is the format of the text displayed under the data set icon that appears when you edit a schema table.

Usually it is just the connector name, that is why the default value is ${connector}, which is just a placeholder for the connector name.

In some cases, however, it is desired to display more details, such as in the file system connector where the file name is also displayed.

Placeholders in the format string are substituted with the connector name (in case of ${connector} placeholder) or data set property names (e.g. ${filename}).

predefined

This flag indicates that the connector instance is created and maintained by Incorta, not by the user. There can be a single connector instance of this type per tenant, and it is named after the subTypeLabel value.

This field is used mainly by local file system connector, because local files are stored in one place that can't be changed.

Tag: connectorProperties

A required container tag for connector properties (propertyModel tags). The order of the properties inside this tag is the same as the order of the property fields as displayed in the UI. This tag has no attributes.

Tag: datasetProperties

A required container tag for data set properties (propertyModel tags). The order of the properties inside this tag is the same as the order of the property fields as displayed in the UI. This tag has no attributes.

Tag: propertyModel

This tag is used to define a single property field that must be displayed in the UI.

Table of supported types for the <propertyModel> tag attributes

TypeDescription
INTEGERA numeric integer field.
BOOLEANA yes/no toggle.
HIDDENNot displayed in the UI. Used when there is a generated property that needs to be maintained.
TEXTA small text field.
MEDIUM_TEXTA medium text field.
LONG_TEXTA long text field (e.g. for SQL queries).
PASSWORDA password field.
LISTA dropdown list where user can select one of predefined values.
LIST_WITH_OTHERSimilar to LIST, except that user can also choose the option "Other" which shows a text field where he enters a text value.
DYNAMIC_LISTSimilar to LIST, except that the values of this list are obtained dynamically from the backend, by invoking either refreshConnectorProperty() in the connector factory, or refreshDataSetProperty() in the connector.
FILEA file upload field.
CATALOG_BROWSERUsed for browsing and selecting a catalog. To navigate catalogs, the UI invokes getCatalogs() in the connector.
SCHEMA_BROWSERUsed for browsing and selecting a schema. To navigate schemas, the UI invokes getCatalogs() and getSchemas() in the connector.
OAUTHOAuth access token. The UI renders an "Authorize" button which opens a new browser tab for obtaining OAuth authorization.

In case a property is of type LIST or LIST_WITH_OTHER, the property must also have a child possibleValues tag to list the property values.

A property can also be dependent on other properties. It can be displayed by the UI only if specific values were selected in other properties. This is done by specifying those dependencies in the child dependencies tag.

Property Attributes

Table of <propertyModel> attributes

AttributeRequiredDefault ValueDescription
typeYesThe property type. See the table above.
nameYesThe name of the property.
labelYesThe property label to be displayed in the UI.
prefixNoStatic text to be displayed before the property field.
suffixNoStatic text to be displayed after the property field.
tooltipNoTooltip text to be displayed next to the property label.
defaultValueNoDefault property value. The UI is pre-populated with this value the first time the user edits the connector or data set.
minValueNoMinimum value for INTEGER property.
maxValueNoMaximum value for INTEGER property.
extensionsNoComma-separated list of file extensions. Used only for FILE properties.
extensionsPropertyNoName of another property, the value of which is a comma-separated list of file extensions to be used by the current property. Used only with FILE properties.
usernamePropertyNoUsed with OAUTH properties to store the name of another HIDDEN property used to store the authenticated user name.
dependsOnPropertiesNoA comma-separated list of property names. When any of these properties are modified, the current property is refreshed by invoking either refreshConnectorProperty() in the connector factory, or refreshDataSetProperty() in the connector. Used for DYNAMIC_LIST, CATALOG_BROWSER and SCHEMA_BROWSER property types.
requiredNofalseA flag that indicates whether or not this property is required.

Tag: possibleValues

A container tag for predefined property values (propertyValue tags). This tag is used only for properties of type LIST or LIST_WITH_OTHER.

Tag: propertyValue

Represents a single predefined value for a property of type LIST or LIST_WITH_OTHER.

AttributeRequiredDefault ValueDescription
valueYesProperty value.
labelYesLabel for property value to be displayed in the UI.

Tag: dependencies

An optional container tag for property dependencies (propertyDependency tags). It is used when we need to make a field appear only when user selects a specific value for another property.

For example, the queryUpdate property that contains the incremental query should be displayed only if the incremental property is set to true.

There can be more than one dependency. The current property will be displayed only if all dependencies are satisfied.

Tag: propertyDependency

Represents a single property dependency.

AttributeRequiredDefault ValueDescription
propertyNameYesName of dependency property to be checked.
propertyValueYesValue of dependency property to be checked.
inverseNofalseWhen set to true, the dependency condition is inverted, which means that the current property will be displayed only when this dependency condition is NOT satisfied. For example, this can be used when we need to check that a field is not empty.