Monitoring predictions
Archived
4 Tasks
15 mins
Beginner
English
Scenario
U+ Bank has recently implemented Pega Customer Decision Hub™ to display a personalized credit card offer to eligible customers on their website. The business requests a report on the boost in success rate that the artificial intelligence (AI) generates. As a data scientist, you inspect the Predict Web Propensity prediction that aims to optimize the click through rate of the web banners that contain the credit card offers.
Use the following credentials to log in to the exercise system:
| Role | User name | Password |
|---|---|---|
|
Data scientist |
DataScientist |
rules |
Caution: This challenge requires specific artifacts. Ensure that you click Initialize (Launch) Pega instance for this challenge to get the correct exercise system.
Your assignment consists of the following tasks:
Task 1: Inspect the control group settings
Inspect the settings for the control group by answering the following questions:
- Is the control group based on a customer attribute?
- Will the control group be large enough to measure the lift?
Task 2: Inspect the success rate graph
Inspect the success rate of the Control group and the Test group over time by answering the followings questions:
- Is the success rate of the Control group stable over time? Is this expected?
- Is the success rate of the Test group higher or lower than the success rate of the Control group? Is this expected?
Task 3: Inspect the lift graph
Inspect the lift over time by answering the following questions:
Task 4: Inspect the performance graph
Inspect the performance of the AI by answering the following questions:
- What is the current performance of the prediction?
- What is the current distribution of the prediction?
- Which of the credit card offers have a low success rate?
Challenge Walkthrough
Detailed Tasks
1 Inspect the control group settings
- Log in as Data Scientist with user name DataScientist and password rules.
- In the navigation pane on the left, click Intelligence > Prediction Studio.
- Click the Predict Web Propensity tile to open the prediction configuration.
| Question - Is the control group based on a customer attribute? |
- Click the Settings tab.
Tip: By default, the Control group is based on a percentage. Based on best practices, the Control group is set to 2% of the customers. Alternatively, you can base the Control group on a customer attribute by selecting Field.
| Question - Will the control group be large enough to measure lift? |
Tip: In a web scenario, with high volumes of customer interaction, 2% is sufficient to measure lift.
2 Inspect the success rate graph
- Click the Analysis tab.
- Inspect the success rate graph.
| Question - Is the success rate of the Control group stable over time? Is this expected? |
Tip: The success rate of the Control group is dependent on customer behavior and engagement policies, so it may change over time.
| Question - Is the success rate of the Test group higher or lower than the success rate of the Control group? Is this expected? |
Tip: The success rate of the Test group is expected to be higher than the success rate of the Control group as AI calculates what credit card offer has the highest likelihood of a click-through. The credit card offer is random in the Control group.
3 Inspect the lift graph
| Question - What is the current boost in success rate that the AI generates? |
- Inspect the lift graph.
Tip: Hover over a date to display the lift. The current lift is also shown in the upper left of the graph: 103.72%.
| Question - What is the trend of the lift? |
Tip: As U+ Bank implemented Pega Decision Management recently, there is not enough data to detect a trend yet. Once sufficient data is collected, the system generates notifications when there is no lift or when the lift drops significantly.
4 Inspect the performance graph
- Inspect the performance graph.
Tip: Hover over a date to display the performance. The current performance is also shown in the upper left of the graph: 84.63 (AUC).
| Question - What is the current distribution of the prediction? |
- Click Show distribution.
Tip: The distribution can be displayed as a ROC curve and a decile distribution for further inspection.
| Question - Which of the credit card offers have a low success rate? |
- In the navigation pane on the left, click Models.
- Click the Web_Click_Through_Rate tile to open the model configuration.
- On the Monitor tab, inspect the performance graph.
Tip: The green StandardCard has a lower success rate than the other cards. It also has a low performance, and inspection of the predictors that the adaptive models use, which drive the Prediction, might improve the prediction lift.