Asynchronous integration
In modern enterprise applications, asynchronous processing is not just a feature; it is an essential architectural requirement for building scalable, resilient, and responsive systems. Pega Platform™ provides a sophisticated set of tools to handle tasks in the background, ensuring that user-facing processes remain fast and fluid while background work executes reliably.
For an architect, choosing the right tool depends on the specific goal of the asynchronous task. These goals fall into three primary architectural patterns:
- UI-centric parallel processing: Shorten perceived wait time for a user on a single screen by running multiple short-lived tasks concurrently.
- Decoupled background processing: Offload a task to run independently of the main process flow, guaranteeing runs without making the user wait.
- Scheduled and recurring Tasks: Run batch jobs or system maintenance activities based on a defined time-based schedule.
UI-centric parallel processing
Improve user experience by loading data or calling services in parallel when a user waits for a screen to render. The entire operation is synchronous from the user's perspective - they cannot proceed until all parallel tasks finish.
Key mechanisms:
- Use the Load-DataPage method (with an Activity)
- Use the Run-In-Parallel option for Connectors (with the Connect-Wait method)
You can use the Call-Async-Activity method to run an activity that uses Load-DataPage method. This loads one or more data pages in separate background threads. The main process continues with other steps until it requires the data, at which point a Connect-Wait step pauses its run until the specified data pages finish loading.
Group multiple Load-DataPage requests by using a shared PoolID. This allows you to wait for all data to be ready with a single Connect-Wait step.
Similarly, most connector methods (for example, Connect-SOAP, Connect-REST) can be configured with the Run-In-Parallel option. This runs the connector in a child thread. The parent activity must then use the Connect-Wait method to join the threads and retrieve the results.
Example use case:
When a customer calls, the dashboard must display the customer's profile, recent case history, and latest order information, each sourced from a different system of record. Instead of calling these services sequentially, design the load sequence to use Load-DataPage with a PoolID to fetch all three data sets simultaneously. The screen renders only after all three API calls complete, significantly reducing total load time.
Architectural Considerations:
- Transaction: Each parallel thread runs in its own context and database transaction. You cannot create a single atomic transaction that spans the parent process and its parallel children. Design error handling accordingly.
- Scope: Use this pattern to improve performance within a single user session. It is not intended for long-running or truly decoupled background work.
- Resource management: Map response data from parallel connectors to separate clipboard pages to avoid race conditions or data overwrites.
Decoupled background processing (modern best practice)
Architectural goal: Achieve true fire-and-forget processing. A user or system process triggers a task that must be run reliably, but the primary workflow does not wait for completion. This approach builds resilient and scalable applications.
Primary mechanism: Queue processors
The modern and architecturally superior approach for decoupled tasks is the Queue Processor. Instead of directly invoking a connector, the originating process uses the Queue-For-Processing method to place a message and associated data onto a dedicated queue. The Queue Processor, running as a multi-threaded background service across the cluster, picks up the message and runs the configured activity.
Architectural advantage over legacy patterns:
- Scalability and high throughput: Queue processors handle high-volume, parallel processing on multiple nodes. Scale horizontally to manage massive workloads more effectively than legacy agents.
- Guaranteed runs and resiliency: Queue processors offer a built-in, configurable retry mechanism. If a transient failure occurs (for example, a downstream API is temporarily unavailable), the Queue Processor automatically retries the operation according to your defined policy before moving it to a broken queue for administrative review.
- Operational visibility: Administrators can monitor throughput, health, and failed items of any Queue Processor directly from Admin Studio.
Example use case:
When a customer submits an online order, confirm the submission and advance the case immediately. The complex, multistep fulfillment process - calling the billing API, updating the inventory system, notifying the shipping department, and sending a confirmation email - should not block the user. Design this by queuing a single Process Order message to a Queue Processor. The Queue Processor orchestrates the back-end calls, retries if necessary, and ensures order processing without impacting user-facing performance.
Architectural considerations:
When designing background logic that may be retried, ensure the operation is idempotent. Running the process multiple times with the same input must yield the same result. For example, Set order status to Fulfilled is idempotent, whereas Add 1 to ShippedItemsCount is not. Design your integration contracts and processing logic to handle retries gracefully.
Scheduled and recurring tasks
Run processes based on a fixed time schedule, not in response to a user action or a specific case. Use this for batch processing and system maintenance.
Primary mechanism: Job schedulers
A Job Scheduler defines a recurring or one-time schedule to run a specific activity. This is the modern replacement for legacy Standard Agents. Job Schedulers are more robust, centrally managed, and provide better visibility into their run status.
Example use case:
Generate a sales report every night at 1:00 AM, synchronize customer data with a central CRM every hour, or archive all resolved cases older than 90 days. Configure a Job Scheduler for each task to ensure automatic implementation without manual intervention.
Legacy patterns (for architectural context)
Older Pega applications or documentation may reference:
- Asynchronous service processing: Configuring a service rule (for example, Service-REST) to queue the incoming request for later processing by the legacy ProcessServiceQueue agent.
- Asynchronous connector processing: Configuring a connector to queue its run through the legacy ProcessConnectQueue agent.
Architectural guidance:
Do not use these legacy patterns for new development. They provide a form of asynchronous fire-and-forget capability but rely on older, less performant Standard Agent infrastructure. For any scenario where these patterns might have been used, choose a Queue Processor instead. Queue Processors provide superior scalability, built-in retry logic, and better administrative oversight.
Check your knowledge with the following interaction: