Architectural use of queue processors and Data Flows
Enterprise applications generate large volumes of business events that require asynchronous processing. Some events require discrete background business action. Other events require continuous processing, enrichment, transformation, aggregation, or routing.
Pega Infinity™ provides two primary processing patterns that can use Stream Service:
- Queue processors
- Data Flows
The two patterns use the same distributed streaming foundation but address different architectural requirements. Understanding these differences helps architects design scalable, maintainable, and operationally efficient event-driven applications. Stream Service provides the distributed event backbone for these asynchronous processing patterns.
Note: Queue processors and real-time Data Flows have the closest relationship to stream processing. Batch and single case Data Flows do not necessarily use Stream Service.
Technical relationship between queue processors and Data Flows
Queue processors and real-time Data Flows can both use the distributed streaming infrastructure in Pega Platform™. However, they provide different application-level abstractions.
A queue processor runs an activity asynchronously for each queued item. A real-time Data Flow processes continuously arriving records through a configurable pipeline of sources, processing components, and destinations.
A queue processor automatically generates a Stream Data Set, which exchanges messages with Stream Service. At run time, Pega Platform creates an in-memory Data Flow work object for the queue processor.
The Data Flow work object performs the following functions:
- Uses the Stream Data Set as its source
- Subscribes to the underlying Apache Kafka topic through the Stream Data Set
- Coordinates the consumption of queued messages
- Sends each message to the activity that the queue processor specifies
The queue processor activity is the destination of this internal run-time Data Flow. A developer configures the queue processor and its processing activity but does not directly create the internal Data Flow work object.
The key distinction is that a queue processor hides the underlying streaming implementation behind a configuration model that focuses on running a business activity for each queued item.
The following figure shows the technical relationship among a queue processor, Stream Service, Data Set, and the real-time Data Flow:
The important architectural insight is that queue processors do not directly consume Apache Kafka topics. Instead, a system-generated Data Flow work object consumes records on behalf of the queue processor.
Note: A queue processor uses the Data Flow runtime internally, but a queue processor is not interchangeable with a developer-authored Data Flow.
The generated Data Flow work object functions as the run-time consumer for the queue processor. It performs the following responsibilities:
- Subscribes to the appropriate Stream Service topic
- Reads messages from the Stream Data Set
- Coordinates consumer processing
- Tracks processing progress
- Manages stream consumption
- Invokes the queue processor activity
The Data Flow work object polls the source Data Set and sends each record to the queue processor activity, which acts as the destination.
Processing models of queue processor and Data Flow
The processing models of both queue processors and real-time Data Flows depends on Stream Service, which acts as the distributed event backbone.
The following figure compares the processing models of a queue processor and a real-time Data Flow:
The following table gives the processing model factors for queue processors and real-time Data Flows:
| Queue processor | Real-time Data Flow |
|---|---|
| Processes discrete queued items asynchronously. | Processes a continuous stream of incoming records or requests |
| Runs the activity that the queue processor configures for each item. | Processes records through configured sources, processing components, and destinations. |
| Supports background work, such as sending a notification, updating another system, or continuing work after the initiating transaction completes. | Supports continuous filtering, transformation, enrichment, decisioning, aggregation, and routing. |
| Uses an auto-generated Stream Data Set and a Data Flow work object internally. | Uses a developer-authored Data Flow with a streamable source. |
| Use a queue processor when an event requires discrete background action. | Use a real-time Data Flow when continuously arriving records must pass through a processing pipeline. |
Check your knowledge with the following interaction:
This Topic is available in the following Module:
If you are having problems with your training, please review the Pega Academy Support FAQs.
Want to help us improve this content?