Processing Real-time Stream Data

In this demo, you learn how to detect patterns in real time from a continuous data stream, for example frequently dropped calls.

A Data Flow reads the call detail records from a data source, feeds the data to an Event Strategy that detects dropped calls, and then writes the output to a destination, which is another Data Set.

At the moment, the data source is a report definition. A report definition is a Pega Platform™ Rule that retrieves and organizes data from various sources and displays it in a user-friendly format. A Data Flow can also read a continuous data stream and feed it to an Event Strategy. Create a copy of this Data Flow, and then make the necessary changes.

The first step is to change the input data source. A stream Data Set provides a means for external systems to send a continuous data stream, which your application can process in real time. So, create a stream Data Set.

To create a Stream Data Set, in the Type list, select Stream.

The class name in the Apply to field is the class that represents the input data.

To expose the Stream Data Set to the outside world, you must define a Pega Rest Service. For this demo, the following artifacts are preconfigured: a Service Package Rule that determines the security scheme and access for the services in the package, a Service REST Rule that exposes a REST API that is externally accessible, and an Activity Rule that saves the incoming data to a Stream Data Set.

The partition key of the Stream Data Set must match the Event key that is defined in the real-time data shape on the Event Strategy. The necessary technical details for both methods are displayed in the Data Set itself.

By default, external applications must comply with basic authentication when sending the data. In this case, the system does not use authentication, so you can turn it off.

Examples of payloads that external applications can send are available in the Data Set for quick reference. Examples come from the properties of the Applies to class of the Data Set. In this case, the payload consists of phone call details.

Now, examine the Event Strategy.

This Event Strategy emits an output when it detects a phone number experiencing three or more dropped calls within a week.

U+ Comms wants to use system time as the event timestamp instead of a property from the data stream. So, set the event timestamp to System time.

The Data Flow writes the output of the Event Strategy to a Data Set configured in the destination.

A real-time Data Flow run must exist to read the continuous data stream from external applications. You can configure the Data Flow on the Real-time processing landing page. For a Data Flow run, the Apply to class is the same as the Apply to class of the Data Flow. Select the Data Flow to set up for the real-time run.

The Data Flow run must be active to receive data from external applications. The Data Flow run is active, so external applications can now send data.

Here is an external REST client that sends data to the Data Flow that you just configured. The payload consists of a phone call detail record according to the example in the Stream Data Set. Make a note of the PhoneNumber and CompletionStatus values.

When the Data Flow receives data, statistics for the processed records update in real time. Send two more dropped call records for the same phone number.

Notice the Data Flow run statistics. Out of the three input records received, the Event Strategy detected the pattern once and then emitted an output record.

Now, examine that record.

As expected, the pattern detected by the Event Strategy corresponds to the phone number 111-1111, which had three dropped calls within a week.

You have reached the end of this video. You have learned:

• How to process a continuous data stream and detect patterns in real time.