Connectors → Splunk

About Splunk

Splunk is a software product that captures, indexes, and correlates real-time, machine-generated data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations. Currently, the Splunk connector extracts data represented as Splunk reports.

Splunk Connector

The Incorta Splunk connector uses the Splunk Software Development Kit (SDK) for Java, which is built as a layer on top of the Splunk REST API. Version 1.0 of the connector supports Splunk reports. The Splunk connector creates a search job to retrieve the list of reports created in the system. When a report is selected during schema design, the Splunk Connector creates another search job to retrieve the fields of that report. Splunk retrieves the fields by discovering them from the last loaded job of the report, a mechanism that works for both scheduled and unscheduled reports.

The Splunk connector supports the following Incorta specific functionality:

FeatureSupported
Chunking
Data Agent
Encryption at Ingest
Incremental Load
Multi-Source
OAuth
Performance Optimized
Remote
Single-Source
Spark Extraction
Webhook Callbacks

Deployment Steps

The Splunk connector is an external connector. You deploy an external connector as a JAR file to each Incorta Node in an Incorta cluster as well as to Cluster Management Console (CMC) host. A System Administrator with root access to the operating systems for each host in the Incorta cluster, including the CMC, will need to deploy the external JAR file for the Splunk Connector. A CMC Administrator will need to restart the Analytics and Loader Services in the cluster. A Systems Administrator will need to restart the CMC.

Deployment to an Incorta Node

Here are the steps to deploy the incorta.connector.splunk.jar file to the extensions directory of an Incorta Node that is running the Analytics and/or Loader Services in an Incorta cluster.

  • Download the Splunk JAR file (incorta.connector.splunk.jar) from the latest version of your Incorta customer release distribution.

  • As the root user for the hosts running Incorta Nodes, use Secure Copy for shell or similar to copy the incorta.connector.splunk.jar to the /tmp directory of the hosts.

    PATH_JAR_FILE='~/Downloads/incorta.connector.splunk.jar'
    INCORTA_NODE_HOST_IPv4_LIST='1.1.1.1 2.2.2.2 3.3.3.3 4.4.4.4'
    PATH_PEM_KEY_FILE='~/.ssh/incorta_2020.pem'
    HOST_ROOT_USER='ec2-user'
    for i in ${INCORTA_NODE_HOST_IPv4_LIST}
    do
    echo $i
    scp -o StrictHostKeyChecking=no -i ${PATH_PEM_KEY_FILE} ${PATH_JAR_FILE} ${HOST_ROOT_USER}@${i}:/tmp/${PATH_JAR_FILE}
    wait
    done
  • Secure shell in to each Incorta Node, and if needed, change the ownership of the file to that of the incorta user.

    sudo su incorta
    sudo chown incorta:incorta /tmp/incorta.connector.splunk.jar
  • For each Incorta Node, as the incorta user, create the splunk directory in the /extensions/connectors/ folder.

    INCORTA_NODE_INSTALLATION_PATH='/home/incorta/IncortaAnalytics/IncortaNode'
    mkdir ${INCORTA_NODE_INSTALLATION_PATH}/extensions/connectors/splunk
  • For each Incorta Node, as the incorta user, move the incorta.connector.splunk.jar from the \tmp file to the splunk directory.

    mv /tmp/incorta.connector.splunk.jar ${INCORTA_NODE_INSTALLATION_PATH}/extensions/connectors/splunk
Restart the Analytics and Loader Services

Here are the steps to restart the Analytics and Loader Services in an Incorta Cluster from the Cluster Management Console (CMC).

  • As the CMC Administrator, sign in to the CMC.
  • In the Navigation bar, select Clusters.
  • In the cluster list, select a Cluster name.
  • Select the Details tab, if not already selected.
  • In the footer, select Restart.

Deployment to the Cluster Management Console

  • Download the Splunk JAR file (incorta.connector.splunk.jar) from the latest version of your Incorta customer release distribution.

  • Using Secure Copy for Shell, copy the incorta.connector.splunk.jar to the /tmp directory of the host running the CMC.

    PATH_JAR_FILE='~/Downloads/incorta.connector.splunk.jar'
    CMC_HOST_IPv4='5.5.5.5'
    PATH_PEM_KEY_FILE='~/.ssh/incorta_2020.pem'
    HOST_ROOT_USER='ec2-user'
    scp -o StrictHostKeyChecking=no -i ${PATH_PEM_KEY_FILE} ${PATH_JAR_FILE} ${HOST_ROOT_USER}@${CMC_HOST_IPv4}:/tmp/${PATH_JAR_FILE}
  • Secure shell into each Incorta Node, and if needed, change the ownership of the file to that of the incorta user.

    sudo su incorta
    sudo chown incorta:incorta /tmp/incorta.connector.splunk.jar
  • As the incorta user, create the splunk directory in the /extensions/connectors/ folder.

    CMC_INSTALLATION_PATH='/home/incorta/IncortaAnalytics/cmc'
    mkdir ${CMC_INSTALLATION_PATH}/extensions/connectors/splunk
  • As the incorta user, move the incorta.connector.splunk.jar from the \tmp file to the splunk directory.

    mv /tmp/incorta.connector.splunk.jar ${CMC_INSTALLATION_PATH}/extensions/connectors/splunk
  • As the incorta user, stop the CMC

    cd ${CMC_INSTALLATION_PATH}
    ./stop-cmc.sh
  • As the incorta user, start the CMC

    cd ${CMC_INSTALLATION_PATH}
    ./start-cmc.sh

Connect Splunk and Incorta

To connect Splunk and Incorta, here are the high level steps, tools, and procedures:

Create an external data source

Here are the steps to create a external data source with the Splunk connector:

  • Sign in to the Incorta Direct Data Platform.
  • In the Navigation bar, select Data.
  • In the Action bar, select + NewAdd Data Source.
  • In the Choose a Data Source dialog, in Application, select Splunk.
  • In the New Data Source dialog, specify the applicable connector properties.
  • To test, select Test Connection.
  • Select Ok to save your changes.

Splunk connector properties

Here are the properties for the Splunk connector:

PropertyControlDescription
Data Source Nametext boxEnter the name of the data source
Authentication Methoddrop down listOptions are:
  ●  Using Splunk Username and Password
  ●  Using AppleConnect
Usernametext boxSplunk Username and Password authentication only
Passwordtext boxSplunk Username and Password authentication only
IdMS Account Nametext boxSplunk AppleConnect authentication only
IdMS Account Passwordtext boxSplunk AppleConnect authentication only
IdMS AppID Keytext boxSplunk AppleConnect authentication only
TOTP Secret Codetext boxSplunk AppleConnect authentication only
Hostnametext boxSplunk hostname
Porttext boxSplunk port

Create a schema with the Schema Wizard

Here are the steps to create a Splunk schema with the Schema Wizard:

  • Sign in to the Incorta Direct Data Platform.
  • In the Navigation bar, select Schema.
  • In the Action bar, select + New → Schema Wizard
  • In (1) Choose a Source, specify the following:
    • For Enter a name, enter the schema name.
    • For Select a Datasource, select the Splunk external data source.
    • Optionally create a description.
  • In the Schema Wizard footer, select Next.
  • In (2) Manage Tables, in the Data Panel, first select the name of the Data Source, and then check the Select All checkbox.
  • In the Schema Wizard footer, select Next.
  • In (3) Finalize, in the Schema Wizard footer, select Create Schema.

Create a schema with the Schema Designer

Here are the steps to create a Splunk schema using the Schema Designer:

  • Sign in to the Incorta Direct Data Platform.
  • In the Navigation bar, select Schema.
  • In the Action bar, select + New → Create Schema.
  • In Name, specify the schema name, and select Save.
  • In Start adding tables to your schema, select Splunk.
  • In the Data Source dialog, specify the Splunk table data source properties.
  • Select Add.
  • In the Table Editor, in the Table Summary section, enter the table name.
  • To save your changes, select Done in the Action Bar.

Splunk table data source properties

For a schema table in Incorta, you can define the following Splunk specific data source properties as follows:

PropertyControlDescription
Typedrop down listDefault is Splunk
Data Sourcedrop down listSelect the Splunk external data source
Report Entry Methoddrop down listSelect an option for specifying the report to create the schema table from:
  ●  Fully qualified name
  ●  Select from list
Report's Fully Qualified Nametext boxThis property appears when the value of Report Entry Method is Fully qualified name. Enter the full name of the report.
Reportdrop down listThis property appears when the value of Report Entry Method is Select from list. Select an available report from the list.
Start Datedrop down listSelect the time window of the report
Full Load Start Datetext boxThis property appears when the value of Start Date is Custom Date. Enter the custom date in yyyy-mm-dd format.
Page Size (in rows)text boxEnter the number of records in a page for the REST API request
CallbacktoggleEnables the Callback URL field
Callback URLtext boxThis property appears when the Callback toggle is enabled. Specify the URL.

Start date options

The start date options apply to unscheduled reports only:

  • Report’s Default Start Time: This option will use the default time window of the report.
  • All Time: This option will run the report to retrieve all available data without restricting the time window.
  • Custom Date: This option allows the user to enter a custom date to get the data from that date.

For scheduled reports, data is extracted from the last load job. In other words, incremental and full loading is supported for unscheduled reports, and full loading only is supported for scheduled reports.

View the schema diagram with the Schema Diagram Viewer

Here are the steps to view the schema diagram using the Schema Diagram Viewer:

  • Sign in to the Incorta Direct Data Platform.
  • In the Navigation bar, select Schema.
  • In the list of schemas, select the Splunk schema.
  • In the Schema Designer, in the Action bar, select Diagram.

Load the schema

Here are the steps to perform a Full Load of the Splunk schema using the Schema Designer:

  • Sign in to the Incorta Direct Data Platform.
  • In the Navigation bar, select Schema.
  • In the list of schemas, select the Splunk schema.
  • In the Schema Designer, in the Action bar, select Load → Load Now → Full.
  • To review the load status, in Last Load Status, select the date.

Explore the schema

With the full load of the Splunk schema complete, you can use the Analyzer to explore the schema, create your first insight, and save the insight to a new dashboard.

To open the Analyzer from the schema, follow these steps:

  • In the Navigation bar, select Schema.
  • In the Schema Manager, in the List view, select the Splunk schema.
  • In the Schema Designer, in the Action bar, select Explore Data.