Predict Web Propensity prediction vs NBA framework
Hi All,
we wanted to check the "Predict Web Propensity" prediction in our Pega Marketing application (v8.4.4) but its Monitoring tab is empty and the "Refresh data" button doesn't help either.
We use a web container(real-time container) configured in NBA designer. If I check the related "Web_Click_Through_Rate" adaptive model then I can find some responses, so in my opinion, this part works well.
To be honest, I would expect that the Predict Web Propensity to work as well, meaning it should capture and show the monitoring data. Could you please confirm this? (maybe I'm totally wrong)
Actually, I tried to debug this in our dev environment and I have some concerns. So if I open Configure -> Decisioning -> Model Management -> Last Responses and click on one of the web responses then the "Prediction" field under the Response Details is blank. And under the Model Details I have 5 items with the following details:
| Business issue | Group | Proposition | Direction | Channel | Status | Rule name | Class |
|---|---|---|---|---|---|---|---|
|
TestIssue1 |
TestGroup1 | {"pyName":"TestAction1","pyTreatment":"TestWeb1"} | Inbound | Web | Updated |
Web_Click_Through_Rate |
Data-Decision-Request-Customer |
Hi All,
we wanted to check the "Predict Web Propensity" prediction in our Pega Marketing application (v8.4.4) but its Monitoring tab is empty and the "Refresh data" button doesn't help either.
We use a web container(real-time container) configured in NBA designer. If I check the related "Web_Click_Through_Rate" adaptive model then I can find some responses, so in my opinion, this part works well.
To be honest, I would expect that the Predict Web Propensity to work as well, meaning it should capture and show the monitoring data. Could you please confirm this? (maybe I'm totally wrong)
Actually, I tried to debug this in our dev environment and I have some concerns. So if I open Configure -> Decisioning -> Model Management -> Last Responses and click on one of the web responses then the "Prediction" field under the Response Details is blank. And under the Model Details I have 5 items with the following details:
| Business issue | Group | Proposition | Direction | Channel | Status | Rule name | Class |
|---|---|---|---|---|---|---|---|
|
TestIssue1 |
TestGroup1 | {"pyName":"TestAction1","pyTreatment":"TestWeb1"} | Inbound | Web | Updated |
Web_Click_Through_Rate |
Data-Decision-Request-Customer |
| TestIssue1 | TestGroup1 | TestAction1 | Inbound | Web | Ignored | OmniAdaptiveModel |
Data-Decision-Request-Customer |
|
—— |
—— |
{"pyModelClassName":"Data-Decision-Request-Customer","pyModelName":"PredictTreatmentPropensity","pyModelRulesetName":"PegaMKT-Engine","pyModelRulesetVersion":"08-04-01"} |
—— |
—— |
Ignored |
—— |
—— |
|
—— |
—— |
{"pyModelClassName":"Data-Decision-Request-Customer","pyModelName":"PredictWebPropensity","pyModelRulesetName":"MA-Artifacts","pyModelRulesetVersion":"01-02-08"} |
—— |
—— |
Ignored |
—— |
—— |
|
—— |
—— |
{"pyModelClassName":"Data-Decision-Request-Customer","pyModelName":"PredictActionPropensity","pyModelRulesetName":"PegaMKT-Engine","pyModelRulesetVersion":"08-04-01"} |
—— |
—— |
Ignored |
—— |
—— |
So as you can see Web_Click_Through_Rate is updated but the rest of the items seem a bit strange(-> lots of missing values).
I also did some research on how web prediction monitoring should work, I mean without NBA framework and found for instance this: https://community.pega.com/knowledgebase/articles/decision-management-reference-materials/creating-and-implementing-response-strategy-predictive-model-monitoring
It says that basically, we need to set the "pyPrediction" property in the response strategy. Ok, it makes sense but I don't find this step in the ootb CaptureInteractionHistory strategy rule. (As far as I know, this is the response strategy in case of web clicks)
Could you pls help me out? Is my initial assumption about web prediction wrong? Or is it a bug and web prediction should also work with NBA framework by default?
Thanks in advance!
Pegasystems Inc.
NL
You are right. pyPrediction is required for Prediction monitoring. We can help you with this. Can you raise a ticket with Pega support so that we can get involved and help you out here.
Updated: 27 Nov 2021 3:53 EST
Saltech Consulting
DE
Let me share the result of the related SR:
Type of Incident:Product
Resolution type: Solution
Root cause: Pegasystems Bug
Root cause description: A defect in Pegasystems’ code or rules. In Pega Marketing 8.4 prediction monitoring cannot be done from within the NBA framework, That feature is not supported in 8.4.x in case of predictions and is introduced only from 8.5 onwards. Even if the pyPrediction property is set by customizing NBA strategies, its value is lost under teh context of the wait processing via the pxWaitForResponses data flow, which is final rule. This is causing no data propagation to the prediction monitoring
Solution type: Explanation
Solution description: This feature has been further improved in 8.5 and we recommend to upgrade to CDH 8.5 or later in order to get this functionality.
Pegasystems Inc.
NL
@AttilaB3 did you check in the Adaptive models landing page if you see processed responses ? If you see recorded responses but no processed responses, select all your models and click Update.
It might be that the responses are not yet processed. OOTB the generated PredictWebPropensity prediction is set up to process responses only after every 5000 records. You can check that in the model settings.
For more info, I recommend go through the https://academy.pega.com/mission/data-scientist/v3 mission, specifically https://academy.pega.com/module/enabling-ai-nba-framework/v2/in/18296
Miki
Saltech Consulting
DE
Hello @MikiMuzsi, thanks for your reply! Yes, if I check the Decisioning -> Model Management landing page then I can find processed responses for the Web_Click_Through_Rate model. I also used the Update button during my tests and it worked well(Recorded Responses -> Processed Responses).
And I modified the settings of this model, the number of responses is 5 (instead of 5000). And pxResponseWaitingTime is 120 (instead of 1800) in order to speed up my testing.
This is how the model looks now:
Note: AUC is still 50, I don't know when it should be recalculated.
And here is my original issue:
The monitoring tab of the PredictWebPropensity prediction is still empty.
I couldn't really debug this issue because as far as I see, the main parts are implemented in java, I mean in the background. Also tried to play with loglevel settings but no luck so far.
Hello @MikiMuzsi, thanks for your reply! Yes, if I check the Decisioning -> Model Management landing page then I can find processed responses for the Web_Click_Through_Rate model. I also used the Update button during my tests and it worked well(Recorded Responses -> Processed Responses).
And I modified the settings of this model, the number of responses is 5 (instead of 5000). And pxResponseWaitingTime is 120 (instead of 1800) in order to speed up my testing.
This is how the model looks now:
Note: AUC is still 50, I don't know when it should be recalculated.
And here is my original issue:
The monitoring tab of the PredictWebPropensity prediction is still empty.
I couldn't really debug this issue because as far as I see, the main parts are implemented in java, I mean in the background. Also tried to play with loglevel settings but no luck so far.
As for the training courses, yeah, I'm aware of these and I've already checked the related challenge: https://academy.pega.com/challenge/measuring-impact-machine-learning/v3/in/18296
Actually, it's a bit strange because if I request a new pega instance for this challenge then this feature works well in v8.6 but in case of v8.4, the monitoring tab is empty, as in my case. I see that v8.6 has some enhancements but in my opinion, the monitoring should also work in v8.4.
Pegasystems Inc.
NL
@AttilaB3 the first screenshot you show, it seems to reflect that your test has worked fine! I do see responses, and some success rate. The AUC is still 50% likely due to a very small nr of responses. So in my opinion the test is successful. Also, when you open the underlying models, can you see the bubble chart (where the adaptive models are shown and their bubbles representing the volume)?
Updated: 21 Oct 2021 2:43 EDT
Saltech Consulting
DE
@Miki Muzsi yes, the chart seems ok as well:
What else do you think I could check? Or should I raise an SR?