Stream service
The stream service is created on Pega Platform™ to enable the asynchronous data flow between Processes. The stream service is a multi-node component based on Apache Kafka. Apache Kafka is a distributed data store optimized for ingesting and processing streaming data in real time. You must enable at least one stream node for the stream service to function.
The stream service ingests, routes, and delivers high volumes of low-latency data, for example, web clicks, transactions, sensor data, work objects, Data Pages, and customer interaction histories. As a resiliency mechanism for operations, the same data is replicated to other nodes in the cluster. Distribution and replication of the stream data records ensure the scalability and fault tolerance of the Stream Service. The service operates as a cluster on one or more servers. You can distribute the servers among multiple data centers to minimize planned or unplanned downtime.
You can use the stream service, for example, to pass correspondence data to delivery channels in Pega Customer Decision Hub™, process customer responses for the Adaptive Decision Manager (ADM), or initiate background processes in any Pega application. Queue processors and Data Flows use stream service to complete their tasks.
Monitor the status of your stream nodes, partitions, disk space, CPU usage, and database availability regularly to keep your stream service operating without errors.
When the stream service operates in external mode, node details are not visible on the stream landing page. Instead, a NORMAL status message appears, indicating that your Pega Platform installation can connect with external Kafka and use it for the stream service. In Pega Cloud® environments, the Pega team configures and maintains your streaming service. You are not required to take any action.
You can perform a simple, manual status check of the stream service on the stream landing page (in the header of Dev Studio, click Configure > Decisioning > Infrastructure > Services > Stream). The stream service relies on the availability of the Pega Platform database, and a slow or unavailable database might seriously impact it.
To understand the health of the Stream service, monitor the queries to the following three databases:
- pr_data_stream_sessions: Represents the currently active Kafka sessions
- pr_data_stream_nodes: Contains all meta-information about the Kafka cluster
- pr_data_stream_node_updates: Keeps the list of recent updates to the pr_data_stream_nodes table. The table does not play a significant role in stream service operations.
If queries exceed one second, think about tuning your database. In the event of planned or unplanned database unavailability, consider restarting your stream nodes, particularly if they appear unhealthy. By default, the stream service maintains two replicas of each record. If you increase the number of stream nodes from two to three or four, adjust the data replication setting to match the number of stream nodes.
The stream service replicates each record across a configurable number of servers. This replication enables automatic failover to these replicas when a server in the cluster fails, ensuring messages remain available despite failures.
Key features of stream service
The following features are available:
- Partitioned data streams
- High-throughput and low-latency streams
- Resilience and fault tolerance streams
- Stateful even processing streams
- Observability and monitoring streams
Partitioned data streams
Data is divided into partitions to allow parallel processing across multiple nodes. This approach supports horizontal scalability and efficient workload distribution.
High throughput and low latency
Data is divided into partitions to allow parallel processing across multiple nodes. This approach supports horizontal scalability and efficient workload distribution.
Resilience and fault tolerance
Built-in replication and checkpointing help prevent data loss in the event of a node failure. Failed events can be retried or routed to Dead Letter Queues (DLQs) for later analysis.
Stateful event processing
The stream service supports windowed aggregations, joins, and real-time analytics. It maintains operator state for advanced scenarios such as rolling averages, anomaly detection, and sessionization.
Observability and monitoring
The Data Flow landing page provides real-time visibility into stream health, throughput, partition lag, and bottlenecks. You can monitor lifecycle events, partition assignments, and error rates to maintain operational reliability.
Advanced use cases of stream service
As organizations adopt real-time architectures, the Pega Platform stream service becomes a strategic enabler for business-critical scenarios. Its distributed, partitioned design and deep integration with data flows allow you to address challenges that were previously difficult or impossible with legacy systems.
With stream service, enterprises can move beyond simple event ingestion to implement sophisticated, stateful processing pipelines that deliver actionable insights, automate complex decisions, and ensure operational resilience. The following advanced use cases illustrate how stream service helps you solve real-world problems with speed, scalability, and reliability:
Real-time fraud detection
Design Data Flows that analyze transaction streams in real time by applying windowed logic to detect suspicious patterns. For example, a Data Flow might aggregate transactions over a 15-minute window and trigger alerts for anomalous activity.
Customer engagement analytics
Stream service enables continuous ingestion of customer interaction data for immediate personalization and response. Data Flows can join real-time events with reference data, enrich profiles, and update dashboards with sub-second latency.
Asynchronous integrations
Decouple system components by publishing events to topics and allowing downstream consumers to process them independently. This pattern improves scalability and fault isolation, supporting microservices and event-driven architectures.
Operational best practices
To help stream service and Data Flows perform reliably at scale, apply proven operational strategies. Effective partition management, error handling, throughput optimization, and security are critical for maintaining robust and compliant data pipelines. The following best practices provide guidance for building resilient and observable solutions in Pega Platform:
Partition key design
Select partition keys that evenly distribute the workload and minimize hotspots. For example, use customer ID or transaction ID instead of static values.
Dead-letter queues
Configure DLQs to capture records that fail after retries. Monitor DLQ growth and establish remediation workflows to address recurring issues.
Idempotency
Ensure that downstream consumers and Data Flow destinations are idempotent to prevent duplicate processing during retries or replays.
Back-pressure management
Monitor partition lag and throughput. Implement throttling or autoscaling to handle spikes in event volume and maintain service levels.
Security and governance
Apply access controls to topics, encrypt sensitive data in transit and at rest, and maintain audit trails for compliance.
Pega Platform stream service empowers you to design and operate real-time, event-driven systems with confidence. By following these practices, you can design and operate real-time, event-driven systems that are scalable, resilient, and transparent.
Check your knowledge with the following interaction: