GenAI monitoring and observability

Performance Analyzer of Logs (PAL) provides a dedicated GenAI Statistics section for analyzing the performance of Agent, Pega GenAI™ Coach, and GenAI™ Connect Rule interactions. These statistics help LSA understand how the application processes GenAI interactions, analyze latency, and identify performance hotspots across GenAI-powered applications. PAL supports applications built on traditional UI architecture and Pega Constellation™, including processing that occurs in Pega Infinity Studio™ and Pega Digital Experience API (Pega DX API) sessions. 

PAL provides statistics such as: 

• Request and response token counts  
• Tokens-per-second values  
• Time to first token  
• Total elapsed time  
• Tool execution counts  
• Time spent in Rule execution, Pega GenAI services, gateway processing, large language model (LLM) processing, tool execution, and data-source processing  
• Initiate Conversation Statistics and Agent Conversation Statistics for Agent and Coach interactions  
• GenAI Connect Statistics for GenAI Connect Rule executions 

For Agent and Coach interactions, the GenAI Statistics section provides Initiate Conversation Statistics and Agent Conversation Statistics. These statistics include metrics related to both internal Pega Platform™ processing and external LLM processing, helping architects understand how the application distributes time across different stages of execution.  

The following figure shows sample PAL statistics for an Agent: 

For GenAI Connect executions, the GenAI Connect Statistics section provides execution metrics such as request and response token counts, total token consumption, and elapsed time for internal and external processing activities. These metrics help architects understand the performance characteristics of GenAI Connect interactions and identify where latency occurs during execution. 

The following figure shows sample PAL statistics for a GenAI Connect Rule: 

PAL results include a GenAI Elapsed value that shows the cumulative time spent on the AI interaction. 

These metrics help architects determine where the application spends time during GenAI processing and whether latency originates from internal Pega Platform processing or external LLM processing.  

For more information, see Monitoring Pega GenAI features performance with PAL. 

Tracer provides dedicated support for debugging Pega GenAI Connect, Coach, and Agent executions. A new GenAI event category in Tracer helps architects filter and analyze events generated during these executions, making it easier to investigate and debug GenAI interactions. 

When tracing GenAI executions, Tracer provides information such as: 

• The beginning and end of Agent runs 
• The beginning and end of data-source loads, including their duration 
• The beginning and end of tool invocations 
• The beginning and end of GenAI Service calls, including response times 
• Time to first token 
• Tokens-per-second throughput 
• A summarized overview of the request payload 
• The sequence of execution steps 

For Agent and Coach interactions, Tracer shows the beginning and end of Agent runs, data-source loads, tool invocations, and GenAI Service calls. Tracer also provides duration and time-to-first-token information, helping architects examine the execution sequence and locate the processing stage associated with an issue. 

The following figure shows Tracer events and detailed execution statistics for GenAI executions: 

For GenAI Connect Rule executions, Tracer provides tokens-per-second throughput, a summarized overview of the request payload, and a clear sequence of execution steps. Architects can use this information to understand how the execution progressed and investigate the event associated with an issue. 

The following figure shows event and response details for a GenAI Connect Rule execution: 

By providing event-level visibility into GenAI processing, Tracer helps architects understand execution sequences, inspect processing steps, and find and fix issues in Agent, Coach, and GenAI Connect Rule executions. 

For more information, see Viewing Tracer results in the Tracer window. 

AI Tracer provides detailed visibility into Agent interactions at run time, helping architects understand how Agents generate responses and perform actions. By tracing Agent runs, architects can analyze Agent behavior, investigate unexpected outcomes, and identify opportunities to improve Agent performance and response quality. AI Tracer supports tracing both active conversations and interactions available in Agent history. AI Tracer is available for Agents in Constellation applications. 

 AI Tracer includes detailed information about Agent interactions, including: 

• Total processing time for an interaction 
• Token usage statistics 
• The AI model used by the Agent 
• Interaction history 
• Information about Agent responses and actions performed during an interaction 
• Detailed information for individual interactions within a conversation 

These insights help architects understand how Agent processing occurs and identify factors that affect Agent behavior and response generation.  

The following figure shows sample AI Tracer output for an Agent interaction: 

In addition to troubleshooting, AI Tracer is a valuable optimization tool when refining Agent and Coach configurations. AI Tracer can help architects: 

• Visualize token usage 
• Identify unnecessary data sent for processing 
• Refine prompts and data mappings iteratively 
• Compare interactions across conversation history 
• Identify opportunities to improve token efficiency and response quality 

AI Tracer helps architects troubleshoot issues, analyze Agent behavior, refine Agent and Coach configurations, and identify opportunities to improve processing efficiency by analyzing Agent interactions, token usage, and conversation history. 

For more information, see Tracing Agents. 

Check your knowledge with the following interaction: