References → Engine Audit
Overview
In releases before 2024.7.x, the Engine audit files track essential information only, including the tenant, the logged-in user, and actions performed, such as viewing or analyzing insights, applying filters, or downloading a dashboard. These details are saved on CSV files in the following location: /home/incorta/IncortaAnalytics/Tenants/<tenantName>/data/audit.
In 2024.7.x, Incorta has introduced a new enhanced version of the audit files. These files capture detailed tracking information, including the request source, service, off-heap memory used for processing the request, and task state time. The new files are saved under the /home/incorta/IncortaAnalytics/Tenants/<tenantName>/engine_audit.2.0 folder.
After upgrading to 2024.7.x, Incorta records user activities in both files by default. However, you can specify one type of them to use. In the Cluster Management Console (CMC) > Clusters > your_cluster > Server Configurations > Analytics Workload Management, for the Analytics Auditing option, specify if you want to log activities in the Standard audit files, the Enhanced ones, or Both.
Note
- To start logging user actions in the audit files, Incorta requires at least one of the actions it tracks, such as opening a dashboard or downloading an insight, to be executed manually or via a scheduled job.
- You can create schemas on top of these files and build dashboards to analyze the recorded details.
In 2024.7.4, a newer version of the enhanced Engine audit, engine_audit.2.1, is available with new audit files and updates to existing ones. After upgrading to this release, audit data will be stored in the /home/incorta/IncortaAnalytics/Tenants/<tenantName>/engine_audit.2.1 folder. If you have schemas reading from the old engine_audit.2.0 folder, create multi-source tables that read from both folders to maintain a complete audit history.
Standard Audit files
User activities recorded in the Standard audit files include:
- Insight actions, such as analyzing or viewing an insight
- Variable evaluation
- Applying filters
- Downloading insights or dashboards
Columns available on these files include the following:
- Date
- Timestamp
- Tenant
- User
- Command
- Elapsed Time (Total duration in seconds)
- Dashboard Layout GUID
- Parameter1 (The insight ID)
- Parameter2 (The dashboard name)
- Parameter3 (The query mode or source: Engine or cache)
- Parameter4 (Filters or prompts)
- Parameter5 (The insight name)
- Parameter6 (The Dashboard ID)
- Parameter7 (The Query duration in seconds)
- Parameter8 (The run ID)
Enhanced Audit files
Folder structure
The engine_audit.2.0 folder contains four sub-folders as follows:
action: Files in this folder include a row for each HTTP request or Scheduler job.action_details: Files in this folder include the action state updates: one row per update.tasks: Files in this folder include a list of tasks included in an action: one row per task. For example, downloading a dashboard tab may contain 10 queries (insights), representing 10 tasks.tasks_details: Files in this folder include the task state updates: one row per update.
In addition to the previous sub-folders, the engine_audit.2.1 folder contains the task_properties folder. Files in this folder record specific properties for each task, such as the number of measures and dimensions in the query.
The task details files in the engine_audit.2.1 folder have two additional columns: rows_count and log_message.
The following section describes the columns in each file.
Action
| Column | Description |
|---|---|
| action_id | The action ID |
| user_name | The name of the user who triggered the action |
| tenant_id | The tenant ID |
| source | The action source: ● PRIVATE_API_RENDER ● PRIVATE_API_DOWNLOAD ● PRIVATE_API_SEARCH ● SQI_X ● SCHEDULER_RENDER ● SCHEDULER_DOWNLOAD ● UNDEFINED (Includes Public API, SQLi, and variable evaluation requests) |
| format | The requested action format: ● JSON ● CSV ● XLSX ● UNDEFINED |
| isSampled | A boolean represented in 1 and 0 options |
| mode | The requested mode from the action: ● VIEW ● ANALYZE |
| dashboard_id | The ID of the dashboard the action is related to. For some actions, such as Analyzer queries, this value is null. |
| dashboard_name | The dashboard name |
| start_time | The action start time |
| end_time | The action end time |
| tasks_count | The number of tasks included in this action |
| termination_state | The action termination state: ● DONE ● FAILED ● ANOTHER_REQUEST_OVERRIDE |
| service_name | The name of the service that executed the action |
Action details
| Column | Description |
|---|---|
| parent_id | The parent action ID |
| state | The action state: ● INITIALIZED ● RUNNING ● PENDING_MATCHING_RUNNING_REQUEST ● SERVED_FROM_MATCHING_REQUEST_RESULT ● REJECTED ● DONE ● FAILED ● ANOTHER_REQUEST_OVERRIDE |
| start_time | The state start time |
| duration | The state duration in Milliseconds |
Tasks
| Column | Description |
|---|---|
| action_id | The parent action ID |
| task_id | The task ID |
| insight_id | The ID of the insight related to the action. For some actions, such as Analyzer queries and searches, this value is null. |
| insight_name | The insight name |
| task_type | The task type: ● QUERY ● SEARCH |
| thread_id | The ID of the thread assigned to execute this task |
| start_time | The task start time |
| end_time | The task end time |
| termination_state | The task terminal state: ● DONE ● FAILED |
Tasks details
| Column | Description |
|---|---|
| action_id | The parent action ID |
| parent_id | The parent task ID |
| state | The task state. For information about the different task states, see Task state details. |
| state_suffix | An identifier for the state if it is repeated in the same task |
| start_time | The state start time |
| duration | The state duration in milliseconds |
| off_heap_usage_bytes | The off-heap memory used while executing the task (Doesn’t include the memory used for loading the columns) |
| rows_count | The number of rows. This is valid only for the following task states: FILTER, MATERIALIZE_FORMULA, GROUP_AND_AGGREGATE, and ADD_EMPTY_GROUPS. |
| log_message | A summary of the filter, for example, ("SALES_B.sales.CUST_STATE_PROVINCE IN_LIST AL,CA,CO,CT,FL,GE,HI,IA,IL,IN,KS,KY,MA,MD... totalSize 107") |
Task state details
The following table describes the different task states, the related task type, and the parent state of each task, if any.
| Task Type | State | Parent State | Description |
|---|---|---|---|
| Query, Search | INITIALIZED | Task initialization and preparation | |
| Query, Search | QUEUED | Task queued and waiting for available running threads | |
| Query, Search | INTERRUPTING | Interruption attempt triggered, but the task not interrupted yet | |
| Query, Search | RENDERING | Rendering the query or search in the required format: JSON, CSV, XLSX, etc. | |
| Query, Search | SERVED_FROM_CACHE | Main query or runtime view retrieved from cache | |
| Query, Search | INTERRUPTED | Task interrupted (timeout, termination, override, etc.) | |
| Query, Search | SUBQUERY_EXECUTION | A general state for all events that occurred during the subQuery execution | |
| Query, Search | FINISH_SUBQUERY_EXECUTION | A general state for all events that occurred while finalizing the subQuery execution | |
| Query, Search | RUNTIME_VIEW_EXECUTION | A general state for all events that occurred during the runtime view execution | |
| Query, Search | FINISH_RUNTIME_VIEW_EXECUTION | A general state for all events that occurred while finalizing the runtime view execution | |
| Query, Search | ACQUIRING_METADATA_LOCKS | Getting read locks for related schemas. This state may take time if a write lock (sync) is acquired on the required schema. | |
| Query, Search | REQUESTING_NEEDED_COLUMNS | Reading columns from files and storing them in memory | |
| Query, Search | CREATE_LAYOUT | A preparation step used by some queries | |
| Query, Search | PROCESSING_SESSION_VARIABLES | Checking and evaluating session variables | |
| Query | FILTER_SUBQUERY | Checking and evaluating the IN Query filter queries | |
| Query | CANT_ACQUIRE_LOCKS_HOLD_FOR_RETRY | Retrying to run the scheduler after timing out due to a sync lock | |
| Query, Search | CHECKING_CACHE | Checking if a cached version of the query is available | |
| Query, Search | TRYING_SQL_EXECUTION | Checking if the query will be executed via the Incorta SQL engine | |
| Query, Search | CANT_EXECUTE_VIA_SQL | Task not executed via the Incorta SQL engine | |
| Query, Search | EXECUTED_VIA_SQL | Runtime view executed via the Incorta SQL engine | |
| Query, Search | RENDERED_VIA_SQL | Main query rendered via the Incorta SQL engine | |
| Query, Search | MATERIALIZE_COLS | Materializing the result set columns in memory after executing the result set query | |
| Query | RENDERED | Query rendering completed | |
| Query, Search | REJECTED | Query rejected because the Analytics task queue exceeded the configured thresholds for Analytics Task Queue Limit Calculation (%) and Analytics Concurrent Service Max Wait Time (seconds), as set in the CMC | |
| Query, Search | DONE | Task completed successfully | |
| Query, Search | FAILED | Task failed | |
| Query, Search | PROCESSING | Task being processed | |
| Query | COMPILE | PROCESSING | Query compilation |
| Query | FILTER | PROCESSING | Applying the query filter per projection |
| Query | MATERIALIZE_FORMULA | PROCESSING | Materializing new columns for runtime formulas used in the aggregate table's dimensions |
| Query | GROUP_AND_AGGREGATE | PROCESSING | Grouping the selected data and performing the required aggregation |
| Query | ADD_EMPTY_GROUPS | PROCESSING | Adding the empty groups to the query. Valid only when the Show empty groups option is enabled for a dimension in a listing or aggregated table. |
| Query | ADD_ROOT_GROUP | PROCESSING | Adding the level 0 group to the query of a listing table |
| Query | FILTER_GROUPS | PROCESSING | Fetching the groups that match the aggregate filter to apply the rollup to, and rolling out the groups that do not match the aggregate filter criteria. |
| Query | ROLLUP | PROCESSING | Calculating totals for each group |
| Query | GROUP_FILTER | PROCESSING | Calculating and applying the aggregated filter |
| Query | MATERIALIZE_MEASURES | PROCESSING | Materializing the measure values, whether listed or aggregated, to the result table after all calculations are done |
| Query | MATERIALIZE_DIMENSIONS | PROCESSING | Materializing the dimension values to the result table after all calculations are done |
| Query | SORT_GROUPS | PROCESSING | Sorting at the group (Dimension) level |
| Query | SORT_DETAILS | PROCESSING | Sorting at the details (Measures) level |
| Query | GRAND_TOTAL | PROCESSING | Calculating the grand total |
| Query | CREATE_RESULT_SET | PROCESSING | Generating the result set after query execution |
| Query | FILTER_CACHE_EVALUATION | PROCESSING | Evaluating a shared filter |
| Query | FILTER_CACHE_WAIT | PROCESSING | Waiting for a shared filter to be served |
| Query | FILTER_SERVED_FROM_CACHE | PROCESSING | Shared filter served from cache |
Task properties
| Column | Description |
|---|---|
| action_id | The action ID |
| task_id | The task ID |
| property | The property key. For information about the different task properties, see Property details. |
| value | The property value |
Property details
The following table shows the available task properties.
| Property | Description |
|---|---|
| AGGREGATED_FILTERS_COUNT | The number of regular aggregated filters |
| AGGREGATED_FORMULA_FILTERS_COUNT | The number of formula aggregated filters |
| INDIVIDUAL_FILTERS_COUNT | The number of regular individual filters |
| INDIVIDUAL_FORMULA_FILTERS_COUNT | The number of formula individual filters |
| HAS_DISTINCT_FILTER | Specifies if the query has a distinct filter |
| MEASURES_COUNT | The number of measures |
| MEASURES_FILTERS_COUNT | The number of regular filters applied to measures |
| MEASURES_FORMULA_FILTERS_COUNT | The number of formula filters applied to measures |
| ROW_DIM_COUNT | The number of row dimensions in a Pivot table |
| ROW_DIM_SORTING_COUNT | The number of regular sorting conditions applied to row dimensions |
| ROW_DIM_FORMULA_SORTING_COUNT | The number of formula sorting conditions applied to row dimensions |
| HAS_ROW_TOTAL | Specifies if the query has a row total |
| COL_DIM_COUNT | The number of column dimensions in a Pivot table |
| COL_DIM_SORTING_COUNT | The number of regular sorting conditions applied to column dimensions |
| COL_DIM_FORMULA_SORTING_COUNT | The number of formula sorting conditions applied to column dimensions |
| HAS_COL_TOTAL | Specifies if the query has a column total |
| SORTING_COUNT | The number of regular sorting conditions |
| FORMULA_SORTING_COUNT | The number of formula sorting conditions |
| HAS_SUB_QUERY | Specifies if the query has a sub-query (result set) |