Question
Future Proof AI
US
Last activity: 31 May 2023 0:27 EDT
Assign Experiment Control Groups strategy
At my current project we want to implement control groups for certain actions.
Eg: Action 1 only 90% of the customers will receive if eligible and Action 2 only 10% of the customers will receive if eligible. The business wants to implement this via Control Group in order to analyse the data in IH and see if the Action 2 get a better response than action 1. We are also planning an upgrade to Pega 8.8 and I saw on the documentation that CDH generates a strategy called "Assign Experiment Control Groups". Is there any configuration in NBA Designer in the Pega 8.8 that will help us to assign different control groups at the action level or the approach is to customize one of the extension point strategies? Any recommendation on control groups will be greatly appreciated.
Pegasystems Inc.
US
Have you considered leveraging the new ‘Impact Analyzer’ feature in v8.8?
In the five Impact Analyzer tests, we use test and control groups to assess value and engagement lift. If you’re trying to gauge click/accept rates, you should look at engagement lift. If you’re trying to gauge the lift in value capture that would be associated with the click/accept, then look at the value lift KPI.
There is a specific test titled ‘How is Next Best Action performing against a random relevant action?’ (see below)
At the top-level, this test shows you the overall value and engagement lift delivered by NBA over a random eligible offer. As you drill down, we are now comparing value capture and engagement rates for that specific action – when NBA decides vs when presented to all the time (effectively). This might be a good way to execute test and control group experimentation in CDH.
If you're simply trying to gauge the click/accept rate of two different actions, you can go to Action Performance Tracker and just select the two actions you want to compare.
Feel free to reach out to me directly if you'd like to discuss further.
Future Proof AI
US
@wongr sorry for the late reply.
We are currently on 8.7. I will love to use the Impact Analyzer when we are on 8.8.
Any idea if there is something OTTB in 8.7 in order to implement a hold out group for an action : eg only 90% of the eligible customers will receive the action? I have an idea to use a Champion Challenger in one of the extension points but I will love to hear other options if there are any.
Cognizant Technology solution
US
@wongr We have the similar requirement. Couple of questions
1. Where do we define control group%. i see there is an option in Impact analyzer. But my understanding is Impact analyzer is not the place from where we send the offer and it will come post we send the offer to test group or control group.
2. How(Which strategy) the control group is selected. Earlier we used to define in segment and run the segment to generate control group customer. What is the current process?
3. Which strategy the control group offer is executed?
4. Can the executed control group be different than when we select the % in impact analyzer?
Victor