Incorta Copilot Audit

Installation Requirements

Before installing this application, you must enable enhanced auditing feature in CMC.

In the Cluster Management Console (CMC) > Clusters > your_cluster > Server Configurations > Analytics Workload Management, for the Analytics Auditing option, specify if you want to log activities in the Standard audit files, the Enhanced ones, or Both.

Physical Schemas

_IncortaCopilotAudit

Business Schemas

_CopilotAudit

Dashboard Folder

Copilot Audit Dashboard

The Copilot Audit Dashboard provides comprehensive monitoring and analysis of Copilot usage, performance, and user feedback. This dashboard consists of the following tabs: Audit and Details.

Audit Tab

The Audit tab offers a complete overview of Copilot usage metrics, query performance, and user feedback.

Insight	Description
KPIs	A set of key performance indicators displaying Last Updated timestamp, # Users, # Sessions, # Queries, and Avg Execution Time (in Sec) for the entire system.
Queries per User Session	An aggregated table showing each user's login name, number of sessions, number of queries, and queries per session ratio.
User Queries by Source with Avg. Execution Time	A table displaying user login names, number of queries, and average execution time broken down by query source (e.g., Copilot Chat).
Queries by Hour of the Day	A line chart visualization showing the distribution of queries throughout the day, with separate lines for number of queries and number of users active at each hour.
Queries by Source	A pie chart showing the distribution of queries by their source (e.g., Copilot Chat).
Queries by Stop Reason	A pie chart displaying the outcome of queries categorized by stop reason (SUCCESS, EXCEPTION, CANCEL) with percentage breakdowns.
Feedback for Queries	A donut chart showing the count and percentage of query feedback, such as "like" reactions.
User Rating	A detailed table showing individual query ratings with columns for Status, Query text, Query Source, Rating, User, Query Timestamp, Category, and Rating Comments.

Details Tab

The Details tab provides a comprehensive audit log of all Copilot interactions with granular information about each query.

Section	Description
Audit Log	A detailed table displaying comprehensive information for each Copilot query with the following columns:
Session Id	Unique identifier for each user session
Query Id	Numerical identifier for each query within a session
Timestamp	Date and time when the query was executed
Login Name	Username of the person who executed the query
User Query	The actual text of the query submitted by the user
Query Source	The interface through which the query was submitted (e.g., Copilot Chat)
Context	Detailed context information including client settings, business schema, table name, and verbosity settings
Tasks	System tasks performed to process the query (e.g., MetadataRetrievalOp, PrepareSqlForInsightOp, SqlExecutorOp, GenerateSqlInsightOp)
Result Summary	The SQL generated by Copilot to answer the user query
Stop Reason	Final status of the query execution (SUCCESS, EXCEPTION, CANCEL)
Copilot Execution Time	Time in seconds taken to process and respond to the query
Debugging Helpers	Additional debugging information if applicable
Feedback Rating	A secondary table showing user feedback on queries with the following columns:
Session Id	Identifier linking feedback to the specific session
Query Id	Identifier linking feedback to the specific query
Feedback Timestamp	Date and time when the feedback was provided
Action	Type of feedback provided (e.g., "like")
Category	Classification of the feedback if applicable (e.g., "Easy to Understand")
Feedback	Specific comments provided by the user

Key Metrics Explained

# Users: Total number of unique users who have used Copilot.
# Sessions: Total number of Copilot sessions initiated.
# Queries: Total number of queries submitted to Copilot.
Avg Execution Time: Average time in seconds for Copilot to process and respond to queries.
Queries per Session: Average number of queries users submit during a single session.
Stop Reason: Categorizes query completions as:
- SUCCESS: Query completed normally
- EXCEPTION: Query encountered an error
- CANCEL: Query was canceled by the user
Feedback: User reactions to query responses (e.g., "like")
Context: Includes information about the business schema, table names, and verbosity settings used for each query
Tasks: System operations performed to process a query, such as metadata retrieval, SQL preparation, and execution
Result Summary: The actual SQL generated by Copilot to fulfill the user's natural language request