Queue processor enhancements for enterprise-scale background processing

Previously, business users had limited visibility into the status of work submitted to a queue processor. Diagnosing failures or reprocessing queue items often required administrative access and specialist support. 

Pega Infinity 26 provides visibility into the queue processor lifecycle. You can monitor queue items as they move through the Scheduled, Enqueued, In Progress, Completed, and Failed states. 

You can inspect failed background operations and retry them from the Work Portal without accessing the administrative portal. A retried queue item uses the execution context of the original attempt, including the applicable data and configuration. You can retry only failed items. You cannot manually reprocess successfully completed items by using this capability. 

Pega Platform™ does not enable progress monitoring by default. To enable monitoring, select the Log for Progress Monitoring checkbox in the relevant background-processing configuration. When monitoring is enabled, Pega Platform persists process-monitoring records in the Log-QueueProcessMonitor class. Recording progress-monitoring information increases database activity and storage use. Enable monitoring only for background processes for which operational visibility provides business value. 

Pega Platform retains process-monitoring records for a maximum of seven days. A scheduled purge removes older records. Therefore, use this capability for short-term operational monitoring and recovery, not for long-term audit retention. 

The following figure shows the process-monitoring configuration for the Run Background automation and the Background Processes page in the Work Portal: 

Enterprise use case 

Consider a loan-origination application in which document validation, compliance screening, and third-party verification services run asynchronously through queue processors. If a verification request fails because an external service is temporarily unavailable, an operations user can inspect the failure and retry the background process from the work portal. 

This capability reduces dependence on platform administrators, supports faster issue resolution, and improves operational resilience. From an architectural perspective, the enhancement brings monitoring and recovery closer to business users while reducing administrative intervention. 

Architects can select the queue processor processing strategy that aligns with the business requirements of an application. The available processing models are: 

• Exactly-once processing 
• At-least-once processing 

New queue processor Rules use at-least-once processing by default. Existing queue processor Rules retain exactly-once processing for backward compatibility. 

The following figure shows the option for configuring exactly-once or at-least-once processing on a Queue Processor form: 

Enterprise use case for exactly-once processing 

Consider a payment-settlement process in which a queue processor posts approved transactions to customer accounts. If a node failure or recovery event causes the same queue item to run more than once, the application might create duplicate financial postings. These duplicate postings can produce reconciliation issues and incorrect business outcomes. 

In this scenario, business correctness takes precedence over throughput. Exactly-once processing provides stronger duplicate-processing protection by maintaining processing offsets in a shared database location. Although this approach increases database activity, the trade-off is appropriate when the application cannot tolerate duplicate processing. 

Enterprise use case for at-least-once processing 

Consider a customer-engagement application that sends promotional notifications after a customer interaction. If a node fails before the Data Flow layer commits the processing offset, another node might process the queue item during recovery and send a duplicate notification. 

For this workload, occasional duplicate processing might be acceptable in exchange for improved throughput and reduced offset-management activity. Applications with idempotent processing or duplicate-detection mechanisms can use at-least-once processing for large-scale communication and integration workloads. 

Selecting a processing strategy 

Select the processing strategy based on the business impact of duplicate processing: 

• Use exactly-once processing when duplicate processing could create an unacceptable business outcome. This mode provides stronger protection at the queue processor level but increases database activity. 
• Use at-least-once processing when the application can tolerate or detect duplicate processing. This mode reduces offset-management activity and can improve throughput. 

The appropriate strategy depends on whether the application can tolerate, detect, or compensate for occasional duplicate processing.

Traditional queue processor alerts rely on static thresholds, which can create excessive operational noise. Before the introduction of baseline-driven alerts, the PEGA0117 alert used configured execution-time thresholds, and the PEGA0134 alert commonly used a static ready-to-process backlog threshold. 

The following figure shows the queue processor option for configuring alert thresholds manually: 

Baseline-driven alerting continuously evaluates normal application behavior and identifies meaningful deviations from established patterns. Pega Platform captures queue processor performance snapshots every 15 minutes and aggregates the snapshots into hourly baselines. Pega Platform generates alerts when execution times or wait times exceed configurable thresholds relative to expected behavior. 

Enterprise use case 

Consider a  customer service application that processes thousands of asynchronous interactions each hour. Backlog levels naturally change throughout the day. Static thresholds can create false-positive alerts during expected peak periods. 

With baseline-driven alerting, Pega Platform evaluates performance against established operating patterns. For example, an unexpected increase in processing latency for a customer-verification queue processor might trigger an alert because the increase differs from normal behavior, not because the latency crossed a generic threshold. 

This behavior produces more actionable alerts and supports faster issue identification. For enterprise support teams, the enhancement shifts monitoring from static threshold management to behavior-based observability. It reduces alert noise and focuses attention on unexpected performance degradation.

Enterprise applications often schedule large volumes of delayed background work. Examples include notification campaigns, Service-Level Agreement activities, policy renewals, and batch integration requests. When many delayed items become eligible at the same time, transferring the items into the processing pipeline can create a bottleneck. 

Pega Platform 26 proactively adjusts the DelayedItemsDataFlowService thread count to increase the rate at which eligible items move from the delayed queue to queue processor topics. 

Pega Platform periodically evaluates upcoming demand, estimates the required processing capacity, and adjusts processing threads before the expected workload increase.  Pega Infinity 26 introduces proactive scaling based on anticipated demand. Every 15 minutes, a Job Scheduler performs the following actions: 

• Counts the delayed items that become eligible during the next 15 minutes 
• Determines the number of active background-processing nodes 
• Reads the baseline average processing time for the applicable activity 
• Calculates the required number of cluster threads 
• Limits the total number of threads to 25 
• Distributes the calculated requirement across the active background-processing nodes 
• Updates the thread count for each node when an adjustment is required 

The maximum of 25 threads corresponds to the 25 partitions used by the delayed-queue table. Pega Platform distributes the System-Message-QueueProcessor-DelayedItem records in the pr_sys_delayed_queue table across 25 logical partitions. These partitions consist of partitions 0 through 23 and a null partition. 

The enhancement anticipates workload increases and adds parallel processing before the items become eligible. This behavior reduces the likelihood that the delayed-item transfer process becomes a bottleneck.

Enterprise use case 

A telecommunications provider might schedule a large number of customer notifications for a specific campaign start time. Similarly, a banking application might schedule overnight compliance reviews, risk assessments, or account-maintenance activities. 

Without proactive scaling, these workloads can experience processing delays because of thread contention. Proactive autoscaling increases the parallelism of the delayed-item transfer service before the items become eligible. This behavior reduces the risk that transferring eligible items to their queue processor topics becomes a bottleneck. 

Proactive autoscaling is useful for event-driven systems with cyclical or predictable workload patterns. Instead of reacting after a backlog forms, Pega Platform anticipates demand and provides additional parallel processing before the expected workload  increases. 

The queue processor enhancements in Pega Infinity 26 expand the operational capabilities of enterprise background-processing architectures. The enhancements address visibility, recovery, processing reliability, observability, and scalability. 

The following table maps common architectural requirements to the applicable enhancements:

For enterprise architects, the primary consideration is aligning background-processing behavior with business requirements. Visibility and self-service recovery improve operational resilience. Configurable processing guarantees support informed trade-offs between duplicate protection and throughput. Baseline-driven alerts improve operational monitoring, and proactive autoscaling supports predictable workload increases. 

Together, these enhancements support scalable background-processing solutions that remain operationally manageable in enterprise environments. 

Check your knowledge with the following interaction: