Skip to main content

Ingesting data in real-time

3 Tasks

20 mins

Visible to: All users
Beginner Pega Customer Decision Hub '23 English
Verify the version tags to ensure you are consuming the intended content or, complete the latest version.

Scenario

U+ Bank uses Pega Customer Decision Hub™ to present personalized credit card offers to its customers on its website.

One of the key attributes when offering a credit-related product to a customer is the "credit score" of the customer. The business team wants to ensure that the customer's credit score is always up to date. So, the team wants to update the Customer Insights Cache in real-time with the latest credit score information.

To achieve this outcome, U+ Bank's IT team developed an integration with the credit bureau to obtain the most recent credit score details for a customer. After the credit score information is available, it is sent to Customer Decision Hub using a REST service.

The IT team shares the following JSON payload:

{
"CustomerID":"14",
"CreditScore":"735",
"CreditScoreUpdatedTime":"20230808T083708.121 GMT"
}

As the System Architect, review the underlying rules to establish the integration. The following artifacts are available for you:

  • A Service Package rule, CreditScoreService, that determines the security scheme and access for the services in the package.
  • A Service REST rule, CreditScore, that exposes a REST API that can be called externally.
  • An Activity rule, WriteToStream, that saves the incoming data to a Stream Data Set.
  • A Stream Data Set, Credit Scores, to store the credit score data received from the service.

As the Decisioning Architect, your role is two-fold:

  • Define a new real-time import data job to update the Credit Score and Credit Score Updated Time properties of the customer whenever a new updated Credit Score is available in the Credit Scores Stream Data Set. Use the data reconciliation feature of a real-time import data job to ensure that the data is only updated if the Credit Score Updated Time is more recent than the value in the Customer Insights Cache.
  • To ensure that only the Credit Score and Credit Score Updated Time properties of the customer record are updated (partial update), extend the data flow created by the real-time data job, and then test the real-time data job using the Credit Score service.
Caution: By default, real-time import data jobs update the full record. All properties of the destination data set are updated with the incoming data from the source Data Set. To make a partial update in the record, create additional processing logic, or filter certain incoming records, extend the pyCustomProcessingDataFlow data flow in the data job class.

Use the following credentials to log in to the exercise system:

Role User name Password
System Architect SystemArchitect rules
Decisioning Architect DecisioningArchitect rules

Your assignment consists of the following tasks:

Task 1: Review the technical artifacts

As the System Architect, review the following artifacts:

Rule type Rule Name

Service Package

Credit Score Service

Service REST

Credit Score

Activity

Write to Stream

Stream Data Set

Credit Scores

Task 2: Define a new real-time import data job

As the Decisioning Architect, use Customer Profile Designer to add the Credit Scores Stream Data Set to the allow list. Then, define a new real-time import data job by using the following details:

Requirement Detail

Data job name

Update Credit Score

Target location

Customer

Source location

Credit Scores

Data reconciliation

Credit score last updated time

Fail a run after more than

1000 records

Task 3: Extend the Data Flow

As the Decisioning Architect, use the pyCustomProcessingDataFlow Sub Data Flow to extend the real-time import data job to ensure that only the Credit Score and Credit Score Last Updated Time fields are updated.

Note: By default, data jobs update the full record. The update might result in data loss when the incoming payload does not include all fields in the destination. To perform a partial update of the target record, use the Sub Data Flow component.

Task 4: Confirm your work

As the Decisioning Architect, send a request using the Service REST rule to validate that the implementation of the real-time import data job is successful.

{
"CustomerID":"14",
"CreditScore":"735",
"CreditScoreUpdatedTime":"20230808T083708.121 GMT"
}

 

You must initiate your own Pega instance to complete this Challenge.

Initialization may take up to 5 minutes so please be patient.

Challenge Walkthrough

Detailed Tasks

1 Review the technical artifacts

  1. On the exercise system landing page, click Launch Pega InfinityTM to log in to Dev Studio.
  2. Log in as the System Architect:
    1. In the User name field, enter SystemArchitect.
    2. In the Password field, enter rules.
  3. In the navigation pane of Dev Studio, click Records > Integration-Resources > Service Package.
  4. In the list of instances, click CreditScoreService to review the Service Package settings.
    Select the CreditScoreService
  5. On the Edit Service Package : Credit Score Service page, in the Context section, review the details:
    1. In the Service access group field, confirm that CDH:Agents is the default selection.
    2. Confirm that the Requires authentication checkbox is clear.
  6. On the Edit Service Package : Credit Score Service form, in the Methods section, confirm that there is a Service REST rule with the Resource path: Insert.
    Verify the settings for Service Package rule
  7. In the Methods section, in the Resource path column, click the Insert to review the Service REST rule.
  8. On the Service REST : Credit Score page, on the Service tab, in the Resource properties section, review the details:
    1. In the Resource field, confirm that UBank-Int-CreditScore is the default entry.
    2. In the Service endpoint URL section, confirm that https://pega.xxxxx.pegaenablement.com/prweb/api/CreditScoreService/v1/Insert is the default URL.
      xxxxx matches the IP address in your browser address line.
      Verify the settings for Service REST rule
  9. On the Service REST: Credit Score form, on the Methods tab, expand POST, and then in the Message data section, review the details:
    1. In the Map to field confirm that JSON is the default entry.
    2. In the Map to key field, confirm that.CustomerPage is the default entry.
    3. In the Mapping method list, confirm that Use fast processing is the default selection.
      Verify the message data
  10. On the Service REST : Credit Score form, on the Methods tab, in the POST section, in the WriteToStream field, click the launch icon to review the Activity.
    Verify the WriteToStream activity
  11. On the Activity: Write to Stream form, on the Steps tab, in the first row, click the arrow, and then in the Method Parameters section, review the following details:
    1. In the Data Set list, confirm that CreditScores is the default selection.
    2. In the Operation list, confirm that Save is the default selection.
  12. Click the open icon on the right of the CreditScores Data Set to see its properties.
    Open the CreditScores Data Set
  13. On the Data set: Credit Scores form, in the upper-right corner, click Actions > Run to browse the Stream Data Set.
  14. In the Run Data Set: Credit Scores window, configure the run:
    1. In the Operation list, select Browse.
    2. In the upper-right corner, click Run.
    3. Confirm there are no records in the stream.
      Preview the CreditScores data set
    4. Close the Data Set Preview window.
  15. Close the Run Data Set: Credit Scores window.
  16. In the lower-left corner, click the user icon, and then click Log off to log out from Dev Studio.

2 Define a new real-time import data job

  1. Log in as the Decisioning Architect:
    1. In the User name field, enter DecisioningArchitect.
    2. In the Password field, enter rules.
  2. In the navigation pane of Customer Decision Hub, click Data > Profile Data Sources to view the data sources in the allow list.
  3. On the Profile Data Sources landing page, click Add > Data set to display the Add Data Set dialog box.
    Add an existing data set
  4. Add the Data Set to the allow list:
    1. In the Add data set window, select Select existing data set, and click Next.
    2. In the Add existing data set window, in the Data set list, select Credit Scores.
      Find the Credit Score data set
    3. In the Event timestamp section, select Use system time.
    4. Click Add to include the data set in the allow-list.
  5. In the navigation pane of Customer Decision Hub, click Data > Data Jobs to view the existing Data Jobs.
  6. On the Data Jobs landing page, click Create import data job to create a real-time import data job:
    1. In the Create import data job: Name & target (1 of 5) window, select Customer.
    2. In the Name field, enter Update Credit Score, and then click Next.
      Select the target data source
    3. In the Create import data job: Source location (2 of 5) window, select Credit Scores, then click Next.
      Select the source location
    4. In the Create import data job: Data reconciliation (3 of 5) window, select the Enable data reconciliation checkbox.
    5. In the Create import data job: Data reconciliation (3 of 5) window, in the Timestamp field, select Credit score last updated time, then click Next.
      Enable data reconciliation
    6. In the Create import data job: Failure policy (4 of 5) window, click Next.
    7. In the Create import data job: Review (5 of 5) window, review the configuration, then click Create.
      Confirm the settings
  7. On the Data Jobs landing page, in the list of data jobs, click Update Credit Score.
  8. On the Data Job: Update Credit Score form, click the Settings tab to access the data flow.
  9. In the Supporting artifacts, click the arrow to extend the section, and then click RTDataFlow_RDI1.

3 Extend the Data Flow

  1. On the Data flow: Data flow for job RDI1 form, double click the Custom Processing Data Flow Sub Data Flow shape.
  2. In the Sub data flow configurations window, in the Data flow field, click the open icon to create a new Sub Data Flow.
    Create a new extension data flow
  3. In the upper-right corner, click Save as.
  4. On the Save As Data Flow: Custom Processing Data Flow form, configure the following details:
    1. In the Apply to field, enter UBank-CDH-Data-Customer.
    2. In the Add to ruleset list, select CDH-Rules.
  5. Click Create and open to create the data flow.
  6. On the canvas, on the first shape, click Add > Merge to add a new merge component.
    Add the Merge component
  7. On the canvas, double-click the unconfigured (- -) component to set up the merge source configurations.
    Configure the Merge component
  8. In the Source configurations window, complete the following settings:
    1. In the Source list, select Data set.
    2. In the Input class field, enter or select UBank-CDH-Data-Customer.
    3. In the Data set field, enter or select Customer.
    4. Click Submit.
  9. On the canvas, double-click the [Merge] component.
  10. In the Merge configurations window, complete the following settings:
    1. In the Merge when all conditions below are met section, in the first list, select CustomerID.
    2. In the Merge when all conditions below are met section, in the second list, enter .CustomerID.
      Enter the merge condition
  11. In the Merge configurations window, in the Default path section, confirm that the Primary path is the default selection, then click Submit.
    Note: For the Primary path, if there are properties with the same name but different values, merge takes the value in the primary source.
    For the Secondary path, if there are properties with the same name but different values, merge takes the value in the secondary source.
  1. In the upper-right corner, click Check in.
  2. In the Check-in comments field, enter any comment, then click Check in.

Confirm your work

  1. In the navigation pane of Customer Decision Hub, click Data > Profile Designer.
  2. In the Customer (Primary) section, click the Customer data source.
  3. On the Data Set: Customer landing page, click the Records tab.
  4. In the list of views, select Credit Score View, then click View.
    Select the Credit Score VIew
  5. Confirm the values for Troy:
    1. In the CustomerID column, confirm that the value is 14.
    2. In the FullName column, confirm that the value is Troy Murphy.
    3. In the Credit score column, confirm that the value is 625.
    4. Confirm that the credit score was last updated at 8/31/2022 6:00AM.
  6. On the Data Set: Customer landing page, click the Data jobs tab, then in the list of data jobs, click the Update Credit Score real-time import data job.
  7. On the Data Job: Update Credit Score landing page, in the Runs section, confirm that the Status column is RUNNING.
    Validate the data job is active and running
  8. In the header of Customer Decision Hub, in the Search field, enter CreditScoreService, then click the Service REST rule to open the rule form.
    Search for the service to make a REST call
  9. On the Service REST: Credit Score landing page, in the upper-right corner, click Actions > Run.
    Run the REST service
  10. In the Simulate REST Service Execution window, complete the following settings:
    1. In the HTTP Method section, select POST.
    2. In the Message Buffer section, enter
      {
      "CustomerID" : "14",
      "CreditScore" : "735",
      "CreditScoreUpdatedTime" : "20230808T083708.121 GMT"
      }
    3. Click Execute.
      Send the JSON Payload using the REST Api
    4. Confirm the Service Simulation Results returns a Success status, and then close the window.
      Confirm the successful result
  1. In the upper-right corner, close the Service REST: Credit Score rule.
  2. On the Data Job: Update Credit Score landing page, in the upper-right corner, click the refresh icon, and then review the details:
    1. In the Runs section, confirm that the Input records column and Successful records columns display 1.
    2. In the Details section, click Customer to confirm the updated credit score.
      Validate the input records
  3. On the Data Set: Customer landing page, click the Records tab.
  4. In the list of views, select Credit Score View, then click View.
  5. Confirm the values for Troy:
    1. In the CustomerID: 14 column, confirm that the value is 14.
    2. In the FullName column, confirm that the value is Troy Murphy.
    3. In the Credit score column, confirm that the value is 735.
    4. Confirm that the credit score was last updated at 8/8/2023 4:37 AM.
      Confirm the final result


Available in the following mission:

If you are having problems with your training, please review the Pega Academy Support FAQs.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best.

Pega Academy has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice