You are viewing content for 4.5 | 4.4 | 4.3 | Previous Releases


Connector SDK guide

The Incorta Connector Software Development Kit, or Connector SDK, allows Incorta customers and partners to develop their own custom connectors.  This is particularly valuable when there is not an out-of-the-box Incorta connector to a particular source system of interest.  The connector SDK provides the framework and steps required to create a new connector that will detect source system metadata, load data in full, and load data incrementally.

With the Connector SDK, you can implement:

  • Metadata discovery

    • Discovering source database tables using schema wizard
    • Discovering table columns while manually editing a schema table
  • Data extraction

    • Execute full load queries on a source system and fetch data records to store in Incorta’s persistence layer
    • Execute incremental load queries on a source system and fetch data records to store in Incorta’s persistence layer

Versions

Important: Ensure you use the correct Connector SDK version with the correct release version of Incorta. See the following table for which version of the Connector SDK you can use with each version of Incorta. You cannot use a different Connector SDK with a different version of Incorta.

Connector SDK Version Incorta Release Version
1.0 4.3, 4.4

Step 1: Implement the Connector

  1. Download the Connector SDK JAR file. The JAR file is shipped with Incorta. You can find it here: <incorta_home>/IncortaNode/runtime/webapps/incorta/WEB-INF/lib/ with the name incorta.connector.api-_<major>.<minor>_.jar.

  2. Implement the following interfaces:

    • com.incorta.connector.ConnectorFactory
    • Main entry point for the connector. It is the factory class used for accessing the connector model and for instantiating the connector.
    • A single factory class can support multiple connector types.
    • com.incorta.connector.Connector
    • Main connector object that manages connection to the data source.
    • Use this parameter to discover source tables using the schema wizard and to create data set objects.
    • com.incorta.connector.DataSet
    • Extracts data for a single Incorta table.
    • It contains the necessary information for extracting the table data. For example, for an SQL-based connector, a data set definition contains the SQL query for that table.
    • A data set can support incremental extraction.
    • com.incorta.util.Record.RecordSet

A record set object represents the set of records returned from a single execution of a data set. Create a record set during the ETL job. The record set is consumed by Incorta’s loader. The record set provides an interface for the loader to iterate over the returned records.

Step 2: Build the Connector

Package the connector into a JAR file. The connector JAR must have a manifest file that:

  • Points to the connector factory class
  • Specifies the connector API version

If the connector requires third-party libraries, you do not need to package them inside the connector JAR. Only deploy these libraries with the connector JAR.

You do not need to deploy the Connector API JAR with the connector and its dependencies, because it is included in your Incorta installation.

The following is a sample connector JAR manifest file. It specifies the connector factory class named SampleConnectorFactory, and indicates that it targets API version 1.0:

Manifest-Version: 1.0
Connector-Factory-Class: com.incorta.connector.sample.SampleConnectorFactory
Connector-API-Version: 1.0

Step 3: Deploy the Connector

Deploy the connector on each Incorta node, and the Cluster Management Console (CMC). The CMC need a connector to access its model, when you import or export tenants.

Deploy the Connector on an Incorta Node

  1. Copy the connector JAR (containing the connector implementation and manifest file) and any third-party library JARs into the following directory: <Installation path>/IncortaNode/extensions/connectors/<connector name>
  2. Restart Incorta services on the node.
  3. (Optional) Create a subdirectory under the connectors directory.

The connector framework scans all directories recursively. A special class loader is used to load classes from the directories to mirror the directory structure. JARs located in separate directories are loaded using separate class loaders. JARs in a subdirectory are loaded using a child class loader.

This is done to ensure isolation between connectors. For example, if one connector uses one version of Apache Commons, while another connector uses a different version of Apache Commons, we want to make sure that each connector code accesses its own correct version.

Deploy the Connector on the CMC

  1. Copy the connector JAR and any dependencies to the following directory: <Installation path>/cmc/extensions/connectors/<connector name>.
  2. Restart the CMC.

Connector Model XML Structure

The connector model XML file is structured as follows: Root element is a <connectorModel> tag, containing one <connectorProperties> tag and one <datasetProperties> tag. Each one of them contains zero or more <propertyModel> tags that describe the properties of the connector or data set.

<?xml version="1.0" ?>
<connectorModel version="1" typeName="sql" typeLabel="SQL Database" subTypeName="mysql" subTypeLabel="MySQL">
   <connectorProperties>
       <propertyModel type="TEXT" name="user" label="Username" required="true" />
       <propertyModel ... />
       ...
   </connectorProperties>
   <dataSetProperties>
       <propertyModel type="LONG_TEXT" name="query" label="Query" required="true" />
       <propertyModel ... />
       ...
   </dataSetProperties>
</connectorModel>

Tags and Attributes

Tag: connectorModel

Root tag of the connector model. The following is a description of its attributes:

Attribute Required Default Value Description
version No 1 The connector version number (not the API version)
typeName Yes Connector type name (e.g. “sql”)
typeLabel Yes Connector type label displayed in UI (e.g. “SQL”)
subTypeName Yes Connector sub-type name (e.g. “mysql”)
subTypeLabel Yes Connector sub-type label displayed in UI (e.g. “MySQL”)
displaySourceFormat No ${connector} This is the format of the text displayed under the data set icon that appears when you edit a schema table. Usually it is just the connector name, that is why the default value is ${connector}, which is just a placeholder for the connector name. In some cases, however, it is desired to display more details, such as in the file system connector where the file name is also displayed. Placeholders in the format string are substituted with the connector name (in case of ${connector} placeholder) or data set property names (e.g. ${filename}).
predefined No false This flag indicates that the connector instance is created and maintained by Incorta, not by the user. There can be a single connector instance of this type per tenant, and it is named after the subTypeLabel value. This field is used mainly by local file system connector, because local files are stored in one place that can’t be changed.
active No false If set to false (default), the connector object will not be created until it is needed. When set to true, the connector object is initialized as soon as it is defined, or right after server startup, and stays available in memory.

Tag: connectorProperties

A required container tag for connector properties (propertyModel tags). The order of the properties inside this tag is the same as the order of the property fields as displayed in the UI. This tag has no attributes.

Tag: datasetProperties

A required container tag for data set properties (propertyModel tags). The order of the properties inside this tag is the same as the order of the property fields as displayed in the UI. This tag has no attributes.

Tag: propertyModel

Represents a single property field that must be displayed in the UI.

A property can have one of the following types:

Type Description
INTEGER A numeric integer field.
BOOLEAN A yes/no toggle.
HIDDEN Not displayed in the UI. Used when there is a generated property that needs to be maintained.
TEXT A small text field.
MEDIUM_TEXT A medium text field.
LONG_TEXT A long text field (e.g. for SQL queries).
PASSWORD A password field.
LIST A dropdown list where user can select one of predefined values.
LISTWITHOTHER Similar to LIST, except that user can also choose the option “Other” which shows a text field where he enters a text value.
DYNAMIC_LIST Similar to LIST, except that the values of this list are obtained dynamically from the backend, by invoking either refreshConnectorProperty() in the connector factory, or refreshDataSetProperty() in the connector.
FILE A file upload field.
CATALOG_BROWSER Used for browsing and selecting a catalog. To navigate catalogs, the UI invokes getCatalogs() in the connector.
SCHEMA_BROWSER Used for browsing and selecting a schema. To navigate schemas, the UI invokes getCatalogs() and getSchemas() in the connector.
OAUTH OAuth access token. The UI renders an “Authorize” button which opens a new browser tab for obtaining OAuth authorization.

In case a property is of type LIST or LISTWITHOTHER, the property must also have a child possibleValues tag to list the property values.

A property can also be dependent on other properties. It can be displayed by the UI only if specific values were selected in other properties. This is done by specifying those dependencies in the child dependencies tag.

Attribute Required Default Value Description
type Yes The property type. See the table above.
name Yes The name of the property.
label Yes The property label to be displayed in the UI.
prefix No Static text to be displayed before the property field.
suffix No Static text to be displayed after the property field.
tooltip No Tooltip text to be displayed next to the property label.
defaultValue No Default property value. The UI is pre-populated with this value the first time the user edits the connector or data set.
minValue No Minimum value for INTEGER property.
maxValue No Maximum value for INTEGER property.
extensions No Comma-separated list of file extensions. Used only for FILE properties.
extensionsProperty No Name of another property, the value of which is a comma-separated list of file extensions to be used by the current property. Used only with FILE properties.
usernameProperty No Used with OAUTH properties to store the name of another HIDDEN property used to store the authenticated user name.
dependsOnProperties No A comma-separated list of property names. When any of these properties are modified, the current property is refreshed by invoking either refreshConnectorProperty() in the connector factory, or refreshDataSetProperty() in the connector. Used for DYNAMICLIST, CATALOGBROWSER and SCHEMA_BROWSER property types.
required No false A flag that indicates whether or not this property is required.

Tag: possibleValues

A container tag for predefined property values (propertyValue tags). This tag is used only for properties of type LIST or LISTWITHOTHER.

Tag: propertyValue

Represents a single predefined value for a property of type LIST or LISTWITHOTHER.

Attribute Required Default Value Description
value Yes Property value.
label Yes Label for property value to be displayed in the UI.

Tag: dependencies

An optional container tag for property dependencies (propertyDependency tags). It is used when we need to make a field appear only when user selects a specific value for another property.

For example, the queryUpdate property that contains the incremental query should be displayed only if the incremental property is set to true.

There can be more than one dependency. The current property will be displayed only if all dependencies are satisfied.

Tag: propertyDependency

Represents a single property dependency.

Attribute Required Default Value Description
propertyName Yes Name of dependency property to be checked.
propertyValue Yes Value of dependency property to be checked.
inverse No false When set to true, the dependency condition is inverted, which means that the current property will be displayed only when this dependency condition is NOT satisfied. For example, this can be used when we need to check that a field is not empty.