Impact Analyzer- NBA vs Random Action
Hi,
I have question with respect to impact analyzer option "How is Next-Best-Action performing against a random relevant action?"
My question is related to the impression of the random action it is showing under Lift details.
when I drill down to each group level or action level the count of impression is changing.
1. How are these Random action's impression is counted?
Can I interpret like below?
if there are 100 times executions of NBA strategy to select a particular action lets say gold card and 5% is in control group does that mean 95 times gold card is selected and 5 time random actions are selected? Similarly if 200 time executions of NBA strategy to select a particular action lets say Platinum card 10 times random action is selected?
2. In all the different issue group related experiment execution
- Are these random action fixed for all customer(That is same random action is given to all customer irrespective of issue group)
- Or for each issue/ group we have random action specific to that issue/group
Pegasystems Inc.
NL
If your control group is 5% this means that for 5% of the customers a random action is chosen *from the eligible actions*. Technically, the model propensity is overridden for that 5%, so it only applies to arbitration. The rules are still in place.
The 5% is based on customers + date, so customers will not be in the control group forever. They will rotate in and out on a monthly basis.
The above means there is no specific "random action". Rather, for that control group, one of the available actions is picked at random rather than decided on by the model.
Hope this helps and cc @wongr for additional comments or references to doc
-Otto
Cognizant Technology solution
US
@Otto_Perdeck Thanks Otto, this clarifies my 2nd question.
With respect to my first question, My doubt still remain How the impressions of random actions are calculated for a specific action(like gold card).
Whenever I select specific action and click on lift details i see the below screenshot
Can I in
terpret like below?
if there are 100 times executions of NBA strategy to select a particular action lets say gold card and 5% is in control group does that mean 95 times gold card is selected and 5 time random actions are selected? Similarly if 200 time executions of NBA strategy to select a particular action lets say Platinum card 10 times random action is selected?
Capgemini
IN
@Otto_Perdeck Does the interaction history have some kind flag to indicate that a particular offer went out based on random selection otherwise the tool is generating the count of impressions and corresponding for random offers over a period?
HCA Healthcare
US
@victorThe Impact Analyzer in Pega measures how well the Next-Best-Action (NBA) performs compared to random actions by tracking how often random actions are shown in the control group. If there are 100 executions for a Gold Card offer with a 5% control group, it means the Gold Card was shown 95 times, and a random action was shown 5 times. Similarly, if there are 200 executions for a Platinum Card, 10 random actions would be shown for the same 5% control setup. When you drill down into specific groups or actions, the impression count changes because it reflects different levels of data, like action-level vs. group-level metrics. Random actions are generally chosen from the same issue/group where the NBA action was selected, meaning for a Credit Card offer, other credit card products may be shown as random actions. However, depending on how the experiment is set up, random actions could be pulled from the entire strategy if multiple groups are involved. This helps compare how effective NBA selections are versus random actions for better decision-making.