Skip to main content

Challenging a predictive model

In the rapidly evolving landscape of data-driven decision-making, organizations are continually seeking ways to enhance their predictive models and ensure optimal outcomes. Learn how to shadow and challenge a Champion model, and ultimately promote the Challenger model to Active status in Pega Customer Decision Hub™.

Video

Transcript

Hi, I'm Bella, a Lead Decisioning Architect working on a Customer Decision Hub project. In this demo, I will show you how to introduce a candidate model for a churn prediction.

U+ Bank uses Customer Decision Hub to optimize customer engagement on the bank's website.

The UBank website

To reduce the number of customers leaving the bank, Customer Decision Hub uses the Predict Churn Propensity prediction. This prediction calculates the likelihood that a customer will switch to a competitor soon. A predictive churn model built on historical data drives the prediction.

The bank personalizes interactions with customers that have a high churn risk, like Troy, and prioritizes retention offers. In this case, Troy receives a 'free extra miles' offer in the hope of retaining him. In contrast, customers with a low churn risk, like Barbara, receive a credit card offer. The predictive power of the churn model declines over time as customer behavior changes, and the model needs to be updated regularly.

You can gently introduce an updated model in a series of deployment cycles. Begin by placing the candidate model in shadow mode, so it does not impact business outcomes. When you submit your changes for deployment, an automatic change request is created in the open revision. The Revision Manager deploys the revision, and the changes become effective in production. Next, you can utilize a Champion Challenger pattern to gradually increase the utilization of the Challenger model instead of the Champion model over time. Lastly, promote the Challenger model to Active status to replace the Champion model.

When you introduce a candidate model, you have the option to validate the candidate model and compare it with the Active model. For this, you need a validation dataset. This validation data set contains the predictor values and the outcome for a set of customers.

Introduce challenger

When introducing a Challenger model, you have different options to select the model. You can use a file-based model, connect to a machine learning service like Google AI or Amazon SageMaker, or select a model that is available in your application.

The source options

To compare the Active and Challenger models, use your validation data set. Optionally, add supporting documents that provide detailed information about the model, its purpose, and its components. These documents help stakeholders understand the structure and functionality of the model, making it easier to maintain, update, and share with others.

Replace the model

The system configures the challenger model in a pending review state.

The challenger pending review

Review the predictive power of the Challenger model compared to the current Champion model, and then approve or reject the Challenger model.

Recommendation

You have the option to shadow, challenge, or directly replace the Active model with the Challenger model.

Option to shadow and challenge

It is recommended to start a Challenger model in shadow mode. In shadow mode, the challenger model runs alongside the current model, receiving production data and generating outcomes without impacting business decisions.

In shadow mode

When you submit for deployment, a change request containing all changes made to the prediction is created in the open revision. When the Revision Manager deploys the revision, the changes take effect in production. Note that your validation data set can be included in the revision.

After running the Challenger model in shadow mode for some time, and the model performs without errors and the range of scores is as expected, you switch to a Champion Challenger pattern.

Start challenging

When you challenge the Active model, in a small percentage of cases the Challenger model is used instead of the Active model.

The distribution

You can edit the distribution to gradually increase this percentage to give the Challenger model more exposure over time and then initiate the promotion of the Challenger model to Active status. By promoting the challenger model, you replace the Champion model with the Challenger model.

Single active model

To introduce a Challenger model, you can also use the available Prediction APIs.

API list

The API provides options to add a model to your application, review and approve or reject a model update, and retrieve the status of a model update.

This allows you to integrate the model approval process in Prediction Studio with any external model deployment process.

  • To summarize:
  • You can shadow the Active model with a candidate model in production.
  • When you submit your changes for deployment, a change request is automatically created in the open revision.
  • When the Challenger model functions as expected in production, you can gradually increase the use of the Challenger model.
  • You can promote the Challenger model to Active status and replace the Champion model.

This Topic is available in the following Module:

If you are having problems with your training, please review the Pega Academy Support FAQs.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best.

Pega Academy has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice