Data Applications → Configure Incorta Data Delivery for Google BigQuery
About Incorta Data Delivery for Google BigQuery
Following is the high-level process Incorta Data Delivery process for Google BigQuery:
- Connect Incorta to your data source with a data application, and load pre-existing physical schema fact and dimension tables.
- Load Synapse data marts from the parquet.
- Visualize data .
Get Started
Following are the steps to get started:
- Review prerequisites.
- Create an Google BigQuery Destination
- Set a Data Destination for a Physical Schema
When naming tenants or data destinations it is best practice to use a generic name. For example, "Incorta" rather than "IncortaDev". Doing so can be helpful when and if you need to import/export tenants or perform data destination integrations.
Create a BigQuery Data Destination
When you create an BigQuery Data Destination, you are able to seamlessly ingest physical schema tables from Incorta to your BigQuery instance.
Here are the steps:
- In the Navigation bar, select Data.
- In the Action bar, select + New → Add Data Destination.
- In the Choose a Data Destination dialog, in Schema Destination select Synapse Analytics.
- In the New Data Destination dialog, specify the following BigQuery schema destination properties:
| Property | Control | Description |
|---|---|---|
| Name your data destination | text box | Enter your data destination name. |
| Project Identifier | text box | Enter the BigQuery project ID. |
| Connection Pool | text box | Enter the maximum number of concurrent tables being sent. The default value is 30. |
| Service Account | text box | Incorta service account that you need to grant the following roles in BigQuery: ● BigQuery Data Editor ● BigQuery Job User For more information, refer to the Google BigQuery documentation. |
- To test the connection, select Test Connection.
- To save, select Ok.
For more information on how to manage Data Destinations, refer to Tools → Data Manager.
Set a Data Destination for a Physical Schema
Set a data destination for physical schemas that will push tables to BigQuery. Here are the steps:
In the Navigation bar, select Schema.
In the Action bar, select Settings (gear icon) → Set Data Destination.
In the Set Schema Destination for <SCHEMA_NAME> dialog:
Select a Schema Destination or select + New Schema Data Destination to create a new one.
Optionally, enter Target Schema Name, which is the table name you want to create in the the data destination. The default value is schema name in Incorta.
NoteIf the schema name already exists in the destination, the newly sent schema will overwrite the existing one.
Optionally, enter a value in the Target Table Prefix to add as a prefix to the table names.
Select Done.
A BigQuery icon will appear on the left of the schema name to indicate the data destination type set for the schema.
Loading data
When you load your schema using full or incremental load, Incorta sends the schema to the data destination automatically. Hence, adding a new stage to the load job details viewer: Send to Destination. For more details, refer to the Load Job Details Viewer.
You can schedule load jobs that trigger data ingest into BigQuery to run in a specific order. For more information, refer to Tools → Scheduler.
If you tried loading multiple schemas in one job and one of them has a data destination set, the stage Send to Destination will always appear.
Limitations
- Encrypted columns are not supported and will be ingested into BigQuery as a string, as it exists in Parquet.