Monitoring GenAI Metrics
As organizations move generative AI capabilities from experimentation to production, Lead System Architects (LSA) face a new challenge: understanding how AI behaves after deployment.
Traditional application monitoring focuses on availability, response times, infrastructure utilization, and transaction failures. These measurements remain important, but they do not show how AI-enabled capabilities interact with users, invoke models, consume tokens, use tools, or affect business processes.
As Pega GenAI™ Agents and Coaches become active participants in enterprise workflows, organizations require operational visibility into AI usage, performance, and user interactions. Without this visibility, LSAs cannot effectively govern AI-enabled systems, investigate issues, optimize behavior, or understand how AI capabilities affect enterprise operations.
Pega Platform™ provides the GenAI Metrics landing page, which presents operational metrics for supported GenAI interactions, models, tools, services, and user activity. GenAI Metrics provides operational data that supports observability, governance, optimization, and continuous improvement of AI-enabled solutions.
For more information, see GenAI Metrics.
Why GenAI Metrics matters
Traditional enterprise applications generally follow predictable processing patterns. When an issue occurs, architects can typically trace the request through application logs, integration points, and background processing components to identify the source of the problem.
Pega GenAI-enabled applications introduce additional complexity. A single interaction might involve multiple model requests, conversation history, contextual information, tool invocations, external services, and user feedback before the application generates a response.
As organizations increase the use of Agents within business processes, understanding how Agents execute those interactions becomes increasingly important. Architects need visibility into how Agents use enterprise resources, how users interact with AI capabilities, and how models, tools, and services process AI-enabled interactions.
Without this visibility, organizations risk treating AI as a black box. When an issue occurs, application teams might have difficulty determining whether the cause originates from the model, Agent design, a tool invocation, a service dependency, or the broader interaction pattern.
GenAI Metrics provides this visibility by capturing and presenting operational information about AI-enabled interactions.
No single metric provides a complete picture of Agent behavior. Effective observability requires multiple measurements that collectively describe how the application processes an interaction, how the interaction consumes resources, and how users experience the outcome.
By bringing these measurements together, GenAI Metrics provides the visibility required to understand how AI-enabled capabilities behave during runtime.
GenAI Metrics insights
GenAI Metrics provides visibility into several aspects of GenAI activity and usage, including:
| Metric and description | Examples |
|---|---|
|
Conversation-related metrics: Provide visibility into conversation volume, user request activity, and how conversations are distributed across channels. |
Total Conversations; Total User Requests; Conversations by Channel |
|
Model metrics: Provide visibility into the models processing requests and trends in model usage. |
Top Models by Requests; Request Trends by Model Over Time |
|
Token metrics: Provide visibility into token consumption associated with GenAI interactions. |
Total Tokens; Request Tokens; Response Tokens |
|
LLM activity metrics: Provide visibility into large language model (LLM) activity. |
Total LLM Calls; token usage trends over time |
|
Tool metrics: Provide visibility into tool utilization and execution activity. |
Total Tool Calls; Top Tools by Execution Count; Tool Execution Status Over Time |
|
Elapsed-time metrics: Provide visibility into the time required to process GenAI requests. |
Aggregated Elapsed Times per User Request; Detailed Elapsed Times per User Request |
|
User feedback metrics: Provide visibility into feedback collected during GenAI interactions. |
User Feedback Overview |
Together, these measurements provide a broader operational view of supported GenAI interactions. By reviewing multiple metrics, application teams can monitor AI-enabled capabilities, investigate operational issues, analyze usage patterns, understand resource consumption, and evaluate user feedback to support informed operational and architectural decisions.
Governance and optimization
Effective governance requires evidence. Use production data rather than assumptions to inform decisions about Agent design, model selection, tool usage, and operational policies.
Operational visibility can support AI governance by providing evidence about supported runtime interactions. GenAI Metrics supplies operational information, while organizations remain responsible for defining and enforcing governance policies.
Runtime visibility enables application teams to move beyond assumptions and make decisions based on operational evidence.
GenAI Metrics supports governance in the following ways:
- Monitor AI adoption
- Identify operational trends
- Investigate unexpected behavior
- Understand resource consumption
- Evaluate Agent interactions
- Review user feedback
- Support optimization initiatives
The capability also supports continuous improvement. As application teams refine prompts, tools, models, and Agent behavior, metrics provide the evidence needed to evaluate whether those changes produce the intended outcomes.
This approach helps organizations improve AI-enabled solutions while maintaining visibility and control over how those solutions operate.
GenAI Metrics in Explore Data
While the GenAI Metrics landing page provides summarized operational insights, organizations often require more detailed analysis.
To support deeper investigation, application teams can add the GenAI Metrics object and its associated metric objects to Explore Data. These objects provide access to detailed feedback, gateway metrics, service metrics, and tool metrics.
These associations provide information about individual interactions for in-depth analysis.
This capability is particularly useful when application teams investigate operational issues, analyze user feedback trends, and perform detailed analysis of GenAI interactions.
For more information, see Adding GenAI Metrics to Explore Data.
LSA considerations
When incorporating GenAI Metrics into an enterprise AI strategy, consider the following architectural practices:
- Identify the Agents and Coaches that require production monitoring.
- Define operational review processes for AI-enabled capabilities.
- Establish appropriate access controls for metric and interaction data.
- Monitor trends instead of isolated measurements.
- Correlate multiple measurements to understand overall behavior.
- Review user feedback with operational metrics.
- Use production evidence to guide optimization decisions.
- Align monitoring practices with organizational governance requirements.
- Include AI observability in the broader enterprise observability strategy.
- Establish baseline usage patterns before using metrics to identify anomalies or optimization opportunities.
GenAI Metrics is more than a reporting feature. It provides the visibility required to govern, operate, and continuously improve AI-enabled applications. Application teams can use operational data to understand and evaluate AI interactions.
Check your knowledge with the following interaction:
This Topic is available in the following Module:
Want to help us improve this content?