Architecture of distributed Stream Service
Modern enterprise applications must handle large volumes of business events while maintaining a fast and responsive user experience. Customer onboarding, Case updates, notifications, integrations, artificial intelligence (AI)-driven actions, and reporting activities often create additional processing requirements that must not delay the primary business transaction. Enterprise architects need architectural patterns that support application scalability without increasing transaction complexity or response times.
Distributed stream processing addresses this challenge by separating business event creation from business event processing. Instead of completing all work immediately, applications publish events that background consumers process independently. This architectural approach supports responsive applications, horizontal scalability, workload-spike management, and failure isolation.
In Pega Infinity™, distributed stream processing uses the Stream Service, which provides the foundation for capabilities such as queue processors and Data Flows. Applications can publish, distribute, and process business events asynchronously across Pega Infinity. This approach helps applications manage high work volumes while maintaining operational resilience and performance.
Stream Service architecture
The Stream Service acts as the intermediary between event producers and event consumers. Producers generate events when business activity occurs, such as when a Case reaches a milestone, an application receives an integration response, or a customer completes an interaction. Producers publish these events to the streaming infrastructure, which stores the events and makes them available to one or more consumers.
Consumers retrieve the events independently and complete the required processing without affecting the original transaction. This separation helps applications scale processing workloads independently of user activity.
At run time, the Stream Service uses a distributed architecture based on Apache Kafka to store and distribute business events for asynchronous processing. The Stream Service distributes events across partitions so that multiple consumers can process workloads concurrently. This partitioning supports increased throughput.
The Stream Service also replicates event data across the stream cluster. Replication supports continued processing when individual nodes become unavailable. As workload demands increase, architects can expand processing capacity by adding infrastructure and consumer resources. This architecture supports enterprise-scale event processing.
The following figure shows how the Stream Service separates business event creation from event processing to support responsive, scalable, and resilient enterprise applications:
Architectural characteristics of the Stream Service
The Stream Service provides architectural characteristics that support scalable, resilient, and high-throughput event processing in enterprise applications.
Decoupling
Event producers and consumers operate independently, which reduces dependencies between business processes and background activities. Applications remain responsive when downstream processing requires additional time or resources.
Example: A customer onboarding Case can complete while the application processes welcome communications, compliance checks, and customer relationship management (CRM) updates asynchronously.
Scalability
Multiple processing resources consume workloads distributed across multiple partitions. As demand increases, these resources process work concurrently.
Example: During a seasonal surge, customers can submit thousands of loan applications while multiple consumers process background verification and integration activities concurrently.
Resiliency
Replicated event data helps maintain processing continuity when individual stream nodes become unavailable. Replication reduces the effect of infrastructure failures on downstream processing.
Example: If a stream node becomes unavailable while processing policy-renewal events, replicated event data remains available to other processing resources.
Throughput
Multiple consumers can process large event volumes concurrently. This concurrent processing helps the application manage substantial asynchronous workloads.
Example: A telecommunications provider can process network alerts, customer updates, and service requests concurrently.
Durability
Events remain available for consumption until processing completes successfully. Durability helps prevent the loss of important business events before processing is complete.
Example: A payment-confirmation event remains available when a consumer is temporarily unavailable. Processing resumes when consumer resources become available.
Operational visibility
Operations teams can use platform observability capabilities to monitor stream health, consumer performance, processing latency, and backlog growth.
Example: Operations teams can identify increasing consumer lag during a peak-processing period and investigate capacity constraints before the constraints affect business service levels.
Lead System Architect (LSA) considerations
When you design enterprise applications, consider the Stream Service as the event backbone for distributed processing in Pega Infinity. The Stream Service supports more than queue processors.
Architectural decisions about asynchronous processing, workload isolation, throughput requirements, failure recovery, and operational monitoring must account for the flow of business events through the Stream Service. A well-designed streaming architecture helps applications remain responsive during peak demand and supports reliability, scalability, and operational control.
Check your knowledge with the following interaction:
This Topic is available in the following Module:
If you are having problems with your training, please review the Pega Academy Support FAQs.
Want to help us improve this content?