Monitoring Gradient boosting models

Hi, I'm Iris, and today I'll show you how to inspect the health of a Gradient boosting model and its predictors in Prediction Studio.

As part of Customer Decision Hub™, the Predict Web Propensity prediction calculates the likelihood that a customer clicks on a web banner to optimize customer engagement.

In this example, the model calculates the propensity that a customer will respond positively to a credit card offer.

It uses the Web Click Through Rate Gradient boosting model configuration to determine the outcomes at the customer level.

The Monitor Models tab shows an intuitive bubble chart that serves as your primary dashboard for assessing model health and identifying optimization opportunities.

The bubble chart visualizes the key metrics of the model at the action-treatment combination level.

Each bubble represents the metrics for a specific offer, in this example a specific credit card in the Web channel, with the size of the bubble reflecting the number of responses, both positive and negative, that have been used in the adaptive learning process.

The Performance axis shows the model performance expressed in the Area Under the Curve, which ranges from 50 to 100; values above 70 generally indicate good predictive performance, with values above 80 representing excellent model accuracy.

The Success rate axis indicates the success rate of the proposition as a percentage, calculated by dividing the number of positive responses by the total number of responses.

This information can help business users and decisioning consultants to refine the Next-Best-Actions.

When you hover the cursor over a bubble, you can view the action name, performance metrics, success rate, and response volume.

The information displayed is extracted from the Adaptive data mart, which is generated automatically by background processes that create snapshots at regular intervals of the Adaptive Decision Management server.

This keeps your monitoring data current and reliable.

The model Context predictors includes channel and direction, enabling the single Gradient boosting model to make contextual splits in its decision trees based on properties such as channel and direction.

From the adaptive model interface, you can drill down into detailed model reports for individual propositions.

The Predictor Importance view shows the relative contribution of each predictor to the model performance as a percentage, providing crucial insights for model optimization.

For example, Annual Income and Credit Score show a high importance score, and strongly contribute to the model's decision-making process.

Inactive predictors such as CLV and Debt To Income Ratio, have an importance score of zero. Gradient boosting models determine predictor usage through their algorithmic decision tree-building process. This information helps data scientists identify which features may need re-evaluation or removal from the model.

The Score Distribution report enables data scientists to validate their model's predictive accuracy and understand how scores are distributed across different customer segments.

The histogram shows response frequencies and propensity percentages across different score ranges, with the data table providing detailed breakdowns of responses, positives, negatives, and lift metrics for each range.

This granular analysis helps identify optimal score thresholds for decision-making.

The trend report visualizes model performance over time, showing how the Gradient boosting model adapts and maintains its predictive accuracy.

This temporal analysis is crucial for identifying performance degradation or improvement patterns.

Regular monitoring of these trends enables proactive model maintenance and optimization.

On the Monitor Predictors tab, you can examine the importance scores that show how each feature contributes to the overall Gradient boosting model performance.

The importance scores reflect how frequently and effectively each predictor is used across the entire ensemble of decision trees.

This detailed predictor analysis enables data scientists to understand which customer attributes drive model decisions, supporting both model transparency and regulatory compliance requirements.

That wraps up this video on the regular monitoring of your Gradient boosting models, ensuring they continue to deliver optimal Next-Best-Actions for your customers.

Until next time!