Predictive performance over time
Hi Team,
We have ADM models which are performing better but sometimes its performance is falling below BAU trend. We were able to see the trend using the ADM model trend report. But want to see the predictor performance over time.
Usecase: Our ADM model has some external model results as predictors along with the fields of Customer Analytic Records. If we can the know predictor performance overtime, we can find the root cause of the external model results and refresh the external models. But by looking at model trend reports, it will only help us to see the overall performance of model. As we know, Pega tries to refresh the predictors over time which will compensate the low performing model predictors with CAR. This will not help us to take a decision on the external model predictors.
Thanks in advance.
Regards,
Nizam
Pegasystems Inc.
NL
@Nizam By default, we don't keep Predictor level snapshots over time. This because it not only contains the predictors of the models but also the binning of those predictors so you can imagine that easily becomes pretty large. There are purging policies in recent product versions but still, it's a lot of data.
You can configure to keep predictor snapshots over time in the configuration of the ADM agent.
There is no OOTB report for a trend chart of the predictors. You would have to build or customize one of the report definitions we ship in the Adaptive category (in the Report Browser), or export the data and use your own BI tools to build such a plot.
Otto
Tech Mahindra
IN
@Otto_PerdeckThanks for sharing the details. Lets say if I change the monitoring option to Store all model data and all predictor data, where can I see this data? If I want to report this either using Pega RD or external tools, what are the table names? Please let me know.
And another question is on the performance of nodes. Does it has any performance impact on changing this setting? Thanks in advance.
Regards,
Nizam
Updated: 25 May 2021 9:51 EDT
Pegasystems Inc.
NL
Hi @Nizam, you can find a description of the tables here https://community.pega.com/knowledgebase/articles/decision-management/database-tables-monitoring-models.
Regarding performance - keeping the predictor snapshots over time will cost space but not performance (the data is pulled in anyway, it just skips first deleting the previous data). The timing is configure in the agent schedule, there is a link in that same config screen.
I'd be interested what use case you are solving with this. Feel free to post back or drop me a note off-line.
Thanks!
Otto
Tech Mahindra
IN
@Otto_Perdeck Thanks for sharing these details. Initially I want to this move data to Teradata and run tableau simulations on top of it. But this might take some time as it involves multiple teams.
As a quick work around, I want to write the predictor bin entry into a log before this is deleted by the agent. It will be moved to elastic search through beats configuration. We have configured Kibana to create simulation on ES index. To do so, I would like to know the rules which gets triggered once the ADM snapshot is captured at the backend. Can we customize this to fit our requirement?
Regards,
Nizam