Question
Newfold
US
Last activity: 26 Jun 2024 7:13 EDT
Random Control Group: Q106. What is being Randomized?
Geek speak: What is the “unit of randomization”? Simply put: What is Pega randomizing “on”? …
Q105a. on PXINTERACTIONIDs? (level 1, very granular, very low)
Q105b. on participants in a single TREATMENT? (level 2, mid-low)?
Q105c. on participants in a CONFIGURATION RULE SET, which may apply to multiple treatments? (level 3, mid-high).
Q105d. on CUSTOMERS (a.k.a., “subject_id”)? (level 4, high level)
In answering, please provide a Pega link that clarifies the topic. The most current Pega link that we see ( Impact Analyzer (pega.com) suggests that the randomization happens on treatment “participants” (level 2). (Here is the same link spelled out: https://docs.pega.com/bundle/customer-decision-hub/page/customer-decision-hub/cdh-portal/impact-analyzer.html )
In the past, however, I believe I saw/heard that Pega was randomizing on customers (level 5).
If there is more than one answer, please clarify which “unit of randomization” applies to the “finalpropensity” scores that (later/downstream) show ups as biased high when they are in their control group status.
Accepted Solution
Updated: 26 Jun 2024 7:13 EDT
Pegasystems Inc.
IN
Please find answers to the questions below,
** Q. Can a subject ID get multiple inbound interactions for the same treatment? That is, can a subject ID get 2 or more impressions for the same treatment? (Believe the answer is,Yes.)
Ans: Yes, it is possible for one subjectID to get multiple inbound interactions for the same treatment unless configured to receive one.
** Q. Let’s assume that a subject ID can have multiple interactions on the same Marketing treatment; Our contact policies sometimes allow 3 impressions (interactions) per customer: Q. Can a given subject ID be in the 5% control group for its first inbound interaction on a Treatment, but be in the 95% test group for its second interaction for that same Treatment?
Ans: This is very unlikely because we use a hashing function (mentioned below) that comprises of the subjectID+year+month as values for the hash and it would be most likely the same for every interaction per subjectID. Although an overlap is possible but it is very unlikely scenario to see such behavior.
Hash function: @hashToProportion(.pySubjectID + @month(@today()) + @year(@today()))
@SethRobinson Could you help us with the behavior of Impact analyzer in splitting up the control group values when it is turned on?
Please find answers to the questions below,
** Q. Can a subject ID get multiple inbound interactions for the same treatment? That is, can a subject ID get 2 or more impressions for the same treatment? (Believe the answer is,Yes.)
Ans: Yes, it is possible for one subjectID to get multiple inbound interactions for the same treatment unless configured to receive one.
** Q. Let’s assume that a subject ID can have multiple interactions on the same Marketing treatment; Our contact policies sometimes allow 3 impressions (interactions) per customer: Q. Can a given subject ID be in the 5% control group for its first inbound interaction on a Treatment, but be in the 95% test group for its second interaction for that same Treatment?
Ans: This is very unlikely because we use a hashing function (mentioned below) that comprises of the subjectID+year+month as values for the hash and it would be most likely the same for every interaction per subjectID. Although an overlap is possible but it is very unlikely scenario to see such behavior.
Hash function: @hashToProportion(.pySubjectID + @month(@today()) + @year(@today()))
@SethRobinson Could you help us with the behavior of Impact analyzer in splitting up the control group values when it is turned on?
** Q. We also lack any IH data elements to see when a given record starts and then ends its (presumed) monthly control group vs. test assignments. Where in the Pega database can we download/export the individual records that are time stamped with the starting date and end date of their control group (e.g., 7/01/24 starts control group vs. 7/31/24 ends control group)? Which specific table names (dimensions or "classes") provide this control group data? And which field names ("properties")?.
Ans: As of today there is no record that gets written to IH that has the information of the control group value.
@Otto_Perdeck Request you to kindly add if I have missed anything here.
Thanks.
Updated: 17 May 2024 6:22 EDT
Pegasystems Inc.
GB
@williamf16809837 please can you clarify - are you quoting a Pega Academy exam? This does not sound like an application issue.
What is the product problem you are needing help with?
This is what our documentation states:
In Pega Customer Decision Hub, the unit of randomization is the subjectID, which typically represents a customer (level 4). This is used in the context of sampling audiences for simulations, where Pega randomly samples up to 20% of your inbound interactions that trigger decision requests. The final inbound sample includes only one random interaction per subjectID. As for the control group in AI models, Pega applies a random propensity to one of the actions for customers in the control group. The 'finalpropensity' scores that show up as biased high when they are in their control group status would also be based on this unit of randomization.
Updated: 4 Jun 2024 10:39 EDT
Newfold
US
@MarijeSchillern Thank you for your response and for the links. I have 2 follow up questions. These are questions that I and others have previously questioned, and are also related to your first link.
** Q. Can a subject ID get multiple inbound interactions for the same treatment? That is, can a subject ID get 2 or more impressions for the same treatment? (Believe the answer is,Yes.)
** Q. Let’s assume that a subject ID can have multiple interactions on the same Marketing treatment; Our contact policies sometimes allow 3 impressions (interactions) per customer: Q. Can a given subject ID be in the 5% control group for its first inbound interaction on a Treatment, but be in the 95% test group for its second interaction for that same Treatment?
To address your questions: I was not referring to a specific exam material. My (our) pain point is that my (our) Pega 8.6 CDH installation does NOT export/report back the sampling fractions (whether 2%? 5%? 12%) that are ACTUALLY BEING IMPLEMENTED inside the CDH brain.
@MarijeSchillern Thank you for your response and for the links. I have 2 follow up questions. These are questions that I and others have previously questioned, and are also related to your first link.
** Q. Can a subject ID get multiple inbound interactions for the same treatment? That is, can a subject ID get 2 or more impressions for the same treatment? (Believe the answer is,Yes.)
** Q. Let’s assume that a subject ID can have multiple interactions on the same Marketing treatment; Our contact policies sometimes allow 3 impressions (interactions) per customer: Q. Can a given subject ID be in the 5% control group for its first inbound interaction on a Treatment, but be in the 95% test group for its second interaction for that same Treatment?
To address your questions: I was not referring to a specific exam material. My (our) pain point is that my (our) Pega 8.6 CDH installation does NOT export/report back the sampling fractions (whether 2%? 5%? 12%) that are ACTUALLY BEING IMPLEMENTED inside the CDH brain.
** We also lack any IH data elements to see when a given record starts and then ends its (presumed) monthly control group vs. test assignments. Where in the Pega database can we download/export the individual records that are time stamped with the starting date and end date of their control group (e.g., 7/01/24 starts control group vs. 7/31/24 ends control group)? Which specific table names (dimensions or "classes") provide this control group data? And which field names ("properties")?.
Pegasystems Inc.
GB
@RAOS4 @Otto_Perdeck @mitra2 are these questions you might be able to help with?
Accepted Solution
Updated: 26 Jun 2024 7:13 EDT
Pegasystems Inc.
IN
Please find answers to the questions below,
** Q. Can a subject ID get multiple inbound interactions for the same treatment? That is, can a subject ID get 2 or more impressions for the same treatment? (Believe the answer is,Yes.)
Ans: Yes, it is possible for one subjectID to get multiple inbound interactions for the same treatment unless configured to receive one.
** Q. Let’s assume that a subject ID can have multiple interactions on the same Marketing treatment; Our contact policies sometimes allow 3 impressions (interactions) per customer: Q. Can a given subject ID be in the 5% control group for its first inbound interaction on a Treatment, but be in the 95% test group for its second interaction for that same Treatment?
Ans: This is very unlikely because we use a hashing function (mentioned below) that comprises of the subjectID+year+month as values for the hash and it would be most likely the same for every interaction per subjectID. Although an overlap is possible but it is very unlikely scenario to see such behavior.
Hash function: @hashToProportion(.pySubjectID + @month(@today()) + @year(@today()))
@SethRobinson Could you help us with the behavior of Impact analyzer in splitting up the control group values when it is turned on?
Please find answers to the questions below,
** Q. Can a subject ID get multiple inbound interactions for the same treatment? That is, can a subject ID get 2 or more impressions for the same treatment? (Believe the answer is,Yes.)
Ans: Yes, it is possible for one subjectID to get multiple inbound interactions for the same treatment unless configured to receive one.
** Q. Let’s assume that a subject ID can have multiple interactions on the same Marketing treatment; Our contact policies sometimes allow 3 impressions (interactions) per customer: Q. Can a given subject ID be in the 5% control group for its first inbound interaction on a Treatment, but be in the 95% test group for its second interaction for that same Treatment?
Ans: This is very unlikely because we use a hashing function (mentioned below) that comprises of the subjectID+year+month as values for the hash and it would be most likely the same for every interaction per subjectID. Although an overlap is possible but it is very unlikely scenario to see such behavior.
Hash function: @hashToProportion(.pySubjectID + @month(@today()) + @year(@today()))
@SethRobinson Could you help us with the behavior of Impact analyzer in splitting up the control group values when it is turned on?
** Q. We also lack any IH data elements to see when a given record starts and then ends its (presumed) monthly control group vs. test assignments. Where in the Pega database can we download/export the individual records that are time stamped with the starting date and end date of their control group (e.g., 7/01/24 starts control group vs. 7/31/24 ends control group)? Which specific table names (dimensions or "classes") provide this control group data? And which field names ("properties")?.
Ans: As of today there is no record that gets written to IH that has the information of the control group value.
@Otto_Perdeck Request you to kindly add if I have missed anything here.
Thanks.