Architectural principles of background processing
In modern Pega applications, background processing has evolved from the legacy agent framework to a robust, scalable architecture driven by queue processors and job schedulers.
As an application architect, understanding the architectural nuances of these Rules is critical. The shift is not merely a change in Rule names; it represents a fundamental change in how the system manages workload, scalability, and throughput.
Processing models
The most significant change for architects is the shift in how work is identified and processed.
- Legacy Agents (pull model): Agents relied on database polling. To find work, an agent periodically woke up and queried the database tables (pr_sys_queues). As data volumes grew, these frequent queries created significant database contention and degraded performance.
- Queue processors (push model): Modern processing uses the Pega Stream Service (based on Apache Kafka). When work is created, it is immediately pushed as a message to a Kafka topic. Consumers (Queue Processors) pick up these messages instantly without stressing the database.
Queue Processors decouple transaction processing from database performance, enabling horizontal scalability.
The following table gives you a quick comparative study between legacy agents and modern background processing methods:
| Feature | Legacy Agents | Modern (queue processor and job scheduler) |
|---|---|---|
| Mechanism | Database polling (pull) | Publisher/subscriber (push) using Kafka |
| Scalability | Limited by database locks & nodes | Horizontal scalability by using threads, nodes, and partitions |
| Performance | High database overhead | High throughput, low database impact |
| Configuration | Complex (agent schedules) | Simplified (rule form and Admin Studio) |
Queue processor: The asynchronous engine
Queue processors replace standard agents and are designed for transactional, high-throughput tasks that must happen now or soon after an event.
Queue processing follows a producer-broker-consumer architecture:
- Producer: An activity calls Queue-For-Processing.
- Broker: The message is published to a Kafka topic (managed by the Stream Service).
- Consumer: The queue processor (running on BackgroundProcessing nodes) subscribes to the topic and runs the logic.
Unlike agents, where you manually tune thread counts, queue processors map threads to Kafka partitions. Pega defaults to 20 partitions for each topic, allowing up to 20 concurrent consumer threads across the cluster without complex configuration.
Job scheduler: The recurring clock
Job schedulers replace advanced agents and are used for recurring tasks such as maintenance, reporting, or batch initiation.
Job schedulers:
- Do not require queuing or Stream Service brokering.
- Wake up, run logic, and go back to sleep.
- Use System Runtime Context, which decouples background jobs from specific operator access groups, simplifying migration and security.
Design decision matrix
Use the matrix below to select the appropriate rule based on architectural needs.
| Design factor | Recommended Rule | Architectural reasoning |
|---|---|---|
| Trigger type | Queue processor | Event-driven: The task is triggered by a specific action (for example, user submits a Case, or an API call is received). Offloading this improves UI response time. |
| Job scheduler | Time-driven: The task is triggered by the clock (for example, daily at 2:00 AM or every 5 minutes). No queuing is required. | |
| Input context | Queue processor | Item-specific: You are processing a specific Case or record (for example, "Process Case ID-123"). |
| Job scheduler | Query-based: You need to query the database to find work (for example, "Find all expired Cases"). | |
| Volume strategy | Queue processor | Horizontal scaling: Can process millions of individual items. Each record is a separate message, allowing multiple nodes to process the file in parallel. |
| Job scheduler |
Batch processing: Suitable for processing large lists query-by-query, often running on a single node to prevent duplication. |
Use case patterns
As an architect, it is important to understand the best use cases for modern background processing methods. These patterns help you make quick decisions for similar implementations in Pega applications:
.
Queue processor use case patterns
- Asynchronous integrations: A user submits a loan application. Use a queue processor to call an external Credit Bureau API in the background, so the user is not blocked waiting for a response.
- Status propagation: When a parent Cases is resolved, use a queue processor to update the status of 50 child Cases in parallel.
- High-volume data parsing: Parse a large CSV file by queuing each row as an individual item for rapid parallel processing.
Job scheduler use case patterns
- SLA management: Query for Cases that have passed their deadline to perform escalation.
- Purge/archive: Run nightly jobs to archive resolved Cases older than one year.
- KPI calculation: Calculate daily aggregate business metrics and write them to a dashboard table.
Critical architectural considerations
When designing background processing solutions, you must account for the following advanced factors to ensure system stability:
- Infrastructure dependency
- Workload isolation (standard versus dedicated)
- Ordering guarantees
- Error handling strategy
Infrastructure dependency
Proper infrastructure configuration is essential for the reliable functioning of queue processors and job schedulers.
- Queue processors require at least one node configured as Stream to manage Kafka topics. If the Stream service is unavailable, queue processors will fail or fall back to a slower database mode.
- Job schedulers require nodes designated as BackgroundProcessing to consume messages and run scheduled jobs.
Workload isolation
Understanding the nature and urgency of tasks helps determine the appropriate queue type:
- Standard queue: Use for generic, low-priority tasks (for example, audit updates).
- Dedicated queue: Use for business-critical workloads. Create dedicated processors for critical logic (for example, PaymentProcessing) to ensure failures or backlogs in shared queues (for example, BulkEmail) do not block critical paths.
Ordering guarantees
Execution order is a key design consideration. Kafka guarantees ordering only within a single partition, not globally.
If Task B must follow Task A, ensure both share the same partition key (commonly the Case ID) so they are placed in the same partition and processed sequentially by the same consumer thread.
Error handling strategy
Defining robust error handling is essential for both front-end and background processes. Queue processors and job schedulers handle errors differently:
- Queue processors: Built-in resiliency. Failed items are retried automatically (up to maximum attempts) and then moved to a broken queue, where they can be reviewed and re-queued by using Admin Studio.
- Job schedulers: Operate on a fire-and-forget model. There is no broken queue. If a job fails, it waits for the next scheduled run. Critical jobs require manual error handling by using Try/Catch logic and notifications embedded in the activity.
Check your knowledge with the following interaction: