Placing a predictive model in shadow mode
Predictive models, built by the U+ Bank data scientist team, drive a prediction that calculates churn risk. The predictive power of these models declines over time as customer behavior changes, and the models need to be updated regularly.
In a business operations environment, a data scientist places the updated model in shadow mode. In shadow mode, the updated model predicts the churn risk on production data, but does not impact the business outcome. This allows the data scientist to monitor the updated model before the team decides to promote the model to active.
Video
Transcript
U+ Bank uses Pega Customer Decision Hub™ to optimize customer engagement on the banks' website.
To reduce the number of customers leaving the bank, Customer Decision Hub uses predictive models, built by the data scientist team, to drive the Predict Churn Propensity prediction that predicts the likelihood that a customer will churn in the near future. Using the churn prediction, U+ personalizes interactions with customers that have a high churn risk and prioritizes retention offers for them.
The predictive power of predictive models declines over time as customer behavior changes. Therefore, the models need to be updated on a regular basis.
To compare the performance of the updated model to the active model, you create a validation data set.
You initiate the update process in the business operations environment.
As a data scientist, you can replace the active model with a predictive model, scorecard, or field in the data model that contains a score.
You can upload a PMML or H2O model to Prediction Studio, connect to a machine learning service, or select a model from the list of available models.
Next, you select the validation data set with which to compare the new model and the currently active model. This analysis provides relevant metrics to help you decide which model performs better on the static data set.
After you evaluate the models, you can approve or reject the candidate model for deployment to production. When you approve the new model, you place the model in shadow mode, which is recommended, or immediately replace the current model with the new model.
If you deploy the candidate model to production in shadow mode, it runs alongside the original model, receives production data, and generates outcomes, but the outcomes do not impact business decisions.
This allows the data scientist to monitor the updated model before the team decides to promote the model to active status.
Approving the model automatically creates a change request in the current revision in Pega 1:1 Operations Manager. As a deployment manager, you deploy the revision that includes the Predict Churn Propensity prediction with the candidate model in shadow mode to the production environment. You have the option to include other rules that the data scientist created in the process, in this case a validation data set to compare the candidate model to the active model.
If testing is not required, you can activate the revision for all users. The updated model now predicts the churn risk on production data, but does not impact the business outcome.
You have reached the end of this video. What did it show you?
- How to place a predictive churn model in shadow mode in the business operations environment.
- How to deploy the revision that includes the predictive churn model in shadow mode to the production environment.
This Topic is available in the following Modules:
If you are having problems with your training, please review the Pega Academy Support FAQs.
Want to help us improve this content?