How can I measure the performance of Predictive models?
When it is a question of how I measure performance of a predictive model used in a strategy , one of some suggestions Pega suggest to use another adaptive model to monitor the performance of a predictive model.
is there any sample strategy diagram or document available to understand this pattern in details?
Regards
Shankha
Pegasystems Inc.
IN
Hi Shankha,
Do you have access to Academy courses in PDN i.e. Pega Decision Consultant course?
Regards
Sibani Saidarsini
Pegasystems Inc.
NL
Hi Shankha,
You can feed the output of your predictive model (for example the 'probability to churn') to an adaptive model as a predictor. The performance of the predictor (AUC) will then be visible in the ADM monitoring.
Please note that you would still need to capture responses (actual outcomes) to obtain an AUC.
In the example below the 'churn rate' output of a predictive model is mapped to a 'ChurnRate' property in the Output Mapping tab.
The adaptive model is then passed the value of 'ChurnRate' as a parameterized predictor.
Ivar
P.S. note that the adaptive model component is in the execution path
Hi Shankha,
You can feed the output of your predictive model (for example the 'probability to churn') to an adaptive model as a predictor. The performance of the predictor (AUC) will then be visible in the ADM monitoring.
Please note that you would still need to capture responses (actual outcomes) to obtain an AUC.
In the example below the 'churn rate' output of a predictive model is mapped to a 'ChurnRate' property in the Output Mapping tab.
The adaptive model is then passed the value of 'ChurnRate' as a parameterized predictor.
Ivar
P.S. note that the adaptive model component is in the execution path
Accenture
IN
HI Ivar,
thanks for the reference model.
in the sample strategy, if I have to use 2 predictive model for comparing performance between those 2 predictive model, then if I have to use champion challenger between those 2 predictive model then how the model will look like?
Regards
Shankha
Pegasystems Inc.
NL
You would have one adaptive model for every predictive model and feed it with just the output of the predictive model as its one and only input.
See also: https://collaborate.pega.com/discussion/tracking-performance-external-models-through-adaptive
Pegasystems Inc.
NL
For completeness sake: If you want to switch between different types or versions of an adaptive model, based on SR properties like e.g. evidence or performance, you would typically use a group-by component, not a switch or champion/challenger
