Discussion
Pegasystems Inc.
IN
Last activity: 4 Feb 2025 2:25 EST
Rethink A/B Testing: How Advanced Multi-Variant Testing Delivers for Diverse Audiences
We will explore the benefits of utilizing multi-variant testing within Pega’s Customer Decision Hub. Our discussion will highlight how Pega’s algorithm continuously learns and optimizes customer experiences in real time—unlike traditional A/B testing, which relies on limited experiments and discards less effective variants. Our approach evaluates each customer individually to determine their propensity to accept each Treatment, leading to an overall increase in engagement rates across all customers.
Watch a video replay...
Download the PDF (see attached)
Find other CDH Community Events like this one
Pegasystems Inc.
IN
Below are the Pending Q&A
1: Does Pega have the capability to generate its own NBC treatments on an ongoing basis to keep that continuous more is more mindset? And then further to that, do you have clients for instance in Financial Services that might not be able to do this due to risk and compliance requirements? Would they need to create all the different options ahead of time?
Ans: Our Intelligent Treatments feature using GenAI to pair messages with images. In a regulated industry we would recommend keeping a human in the loop or creating the options ahead of time in order to mitigate risk.
2: How does conversation modelling support omni channel interactions?
Ans: When configuring conversation modeling in Pega you set up your attribution model.
3: How do you address potential sample ratio mismatch between all of the different options that are included?
Ans: The propensities are normalized in order to compare them to all the other competing models.
4: Do you use Bayesian of Frequentist significance testing in the decisioning engine to find a winning variant? And does a variant need to reach 95% significance and 80% power before a decision is made?
Below are the Pending Q&A
1: Does Pega have the capability to generate its own NBC treatments on an ongoing basis to keep that continuous more is more mindset? And then further to that, do you have clients for instance in Financial Services that might not be able to do this due to risk and compliance requirements? Would they need to create all the different options ahead of time?
Ans: Our Intelligent Treatments feature using GenAI to pair messages with images. In a regulated industry we would recommend keeping a human in the loop or creating the options ahead of time in order to mitigate risk.
2: How does conversation modelling support omni channel interactions?
Ans: When configuring conversation modeling in Pega you set up your attribution model.
3: How do you address potential sample ratio mismatch between all of the different options that are included?
Ans: The propensities are normalized in order to compare them to all the other competing models.
4: Do you use Bayesian of Frequentist significance testing in the decisioning engine to find a winning variant? And does a variant need to reach 95% significance and 80% power before a decision is made?
Ans: The model output is a Bayesian classifier based on the highest propensity. You can set a minimum threshold for the decision to use an adaptive model, if none is set the winning decision will be the model with the highest combination of propensity/value/context/lever (all configured in Pega New Best Action Designer).