CDH Community Event: Adaptive Modeling Lessons from the Field
The predictions functionality in Pega Customer Decision Hub makes it easier than ever to drop adaptive models into Next-Best-Action decision logic. Some work is still needed, however, to help clients leverage the best of what CDH has to offer while providing confidence in the results.
In this session, Cheri Gaudet, Lead Consultant, CXForward shares lessons from the field on how to guide customers through an adaptive modeling project. You'll also learn how you can add value to your project by using OOTB reports.
Watch a replay
Note that Q&A from the session have been posted as replies below. Please continue the discussion there!
View the presentation slide deck (attached)
@shiss In 8.5 and especially 8.6 the UI setting can be easily used to set up a conversion model. Some prior data science discussion can help identify whether a conversion modeling is the appropriate solution, benefits, cons etc. But the technical implementations are made much easier in 8.6.
@shiss Usually we have up to several hundred potential predictors. It is always recommended to use predictors of various sources (demographic data, behavioral data, contextual data). That means, we COULD have up to several hundred but that does not mean we SHOULD have that many, It is better to have a good selection of predictors.
@shiss Some data transformations could be done in the strategy. Type conversion, duration extraction (from date time) and other instance-level transformation could potentially be done on strategy. More complex transformation (Imputing, etc.) should be done outside of the strategy. Some basic imputing could also be done in strategy too (like simply replacing a NULL value with 0 or things of that nature that do not require very sophisticated statistical calculations).
@sundmPega ADM treats date/time fields as "duration" automatically (in recent releases). You can set the "treatment" for those predictors in the predictor section of the ADM rule. Missing value imputation is not usually done nor needed as missing values are treated separately anyway, and the ADM model execution does not impose any restrictions on the data it can process. The best example of feature engineering for ADM models is in pre-processing lists of items - e.g. providing the mean value, the number of items and such "flattening" operations. This should be done outside of the strategy. Going forward, CPD will be the place to define such aggregates.
@shiss A new generation of adaptive models will be available soon. These models are based on Gradient Boosting algorithms (a type of ensemble methods). The new algorithm was released in 8.6 but in lab mode. A newer version will be available soon.
@shiss IH aggregations are extremely powerful. The idea is simple, prior behavior of customers are potential predictors in defining their future behavior. The OOTB IH aggregation dimensions in Pega ADM models have been decided after rigorous testing.
@sundmIf the question is about the number of "records" be aware that this concept is not relevant to an online classifier like ADM is. The data is streamed in and the models (classifiers) are updated in real time. The "memory settings" of ADM refer to the relative weight placed on new responses (by default all responses are weighed equally but that can be changed) but have no impact at all on the physical memory used.
@shiss As of 8.5 we have a new feature to capture the data that passes through ADM. This provides an opportunity to explore the data as well as using it to make new models. This data collection will not just be the customer behavior data, it will include any data that was passed through the ADM rule as well as the response label.
@shiss Model context refers to the context in which the model is making a prediction. A context that includes a treatment means that an adaptive model exists for every treatment in the system. A context that includes only action means that a model exists for every action, which is used for all treatments associated with that action.
@shiss The control group that is configured on predictions allows you to calculate lift (for example in CTR) by providing a benchmark performance to compare against. Champion challenger functionality has been used to accomplish the same thing, but now it is much more straightforward to set up in the prediction.
@shiss Response timeout is a mechanism that provides a solution in outbound. The responses are queued for a certain timeframe, and then they are sent back to the models.
Extend the response timeout window to an appropriate length so that you can capture those later responses. If propensities are low across the board, that’s not completely unusual. Propensities are often very low numbers.
@shiss I don’t think it’s a question of which is better but rather how adaptive models can enhance what they are already doing with their predictive models. There may be good reason to use both. Additionally, the true value of Naive Bayes in Pega is its capability of learning on the fly. Not every off-the-shelf machine learning algorithm has the capability of turning into an online learner. Also, Naive Bayes in Pega is preceded with a data analysis step that prepares the data for the algorithm including taking care of correlated predictors.
@sundm True. To add to that:
- In addition to the algorithm aspect, models built off line often have access to a wider set of data - which is an advantage. However they typically do not have access to all the contextual data in Pega - and context typically is very important for customer behavior. In addition, they can get stale as they're typically developed on older snapshots of the data and need manual updates when new actions/treatments are developed.
- A good way to incorporate them is to include their scores in the XCAR and feed them as (parameterized) predictors to the ADM models. Sometimes they do add value for some specific issues or groups (e.g. acquisition of new customers - where we know little about the customers yet). But often they do not show the same results on real data as when they were developed off-line. - We're working on an even better online algorithm based on XGBoost. Available already like mentioned before but in lab mode.