Question
Merkle Inc.
GB
Last activity: 30 Nov 2020 7:58 EST
How can we use Adaptive model success rate in decision strategies?
The adaptive model provides three parameters as an output Propensity, Performance, and Evidence, but we can see the Adaptive model success rate from the Monitoring page.
We want to use the success rate of the Adaptive models from champion/challenger strategy that contains a different model definition. Is it possible to get the Adaptive Model Success rate as an output in the strategy?
Pegasystems Inc.
NL
No you can't use performance for that, performance is something very different from success rate.
Performance tells you something about the predictive performance of the model. How well is the model able to distinguish people who will click from people who won't, or who will accept vs reject.
The success rate is the term we use for overall success of the action/proposition in a particular channel. So not individually but overall. For example, an e-mail offer for a mortgage might have a low success rate while a banner for free roaming might have a much higher success rate. Performance for the mortgage offer could still be high - it might be possible to identify those few people that are interested.
The overall success rate is not available as an output of the adaptive models. But in recent versions you do get the # of positives next to the evidence, so success rate = positives/evidence is trivially calculated. For older product versions you could work around using a separate adaptive models w/o any predictors so propensity equals overall success rate.
However: can you elaborate on your use case? Switching between models based on success rate is not a usual pattern. You may end up making selections based on base propensities, which you definitely don't want to do as you loose out on the personalization.