I am looking for some guidance on the best practice / way to define impressions as a outcome for adaptive in digital channel. In our application we have configured impressions as negative and click as positive, the problem we are facing is that the ratio of impression to click (negative to positive) is huge because as per our business rules we show proposition X number of times before supressing it.
So the volume of impression can go X times the volume of click. Due to this the ratio between negative to positive is huge and the propensity is very low and model performance is also not good.
I want to know how can we balance the ratio between positive and negative outcome? Do we really need to try and balance the ratio or Pega takes care of it internally? If so how can we overcome the problem of low propensity?
Also, we tried another approach to reduce the negative to positive response ratio by grouping multiple impressions before feeding it to adaptive model, here we are grouping X impression and writing it to Adaptive model once.
These approach is taken to bring down the count of negative responses and have only one negative response for X impressions. I want to understand if this approach is going to be fine or will result in some problem when we upgrade our system to 8.5 from current 7.3 version.
@RohitJ92 Hi Rohit, the performance measure we're using (AUC) is not very sensitive to an imbalance of positives / negatives. Low success rates are common especially in some channels and this is't a problem per se for the adaptive models. You do not need to apply any special tricks to group impressions (but if you do, make sure to do it consistently so you don't introduce bias).
In 8.5+ the way to send impressions (negatives) to adaptive is different but that would not materially impact your numbers. In 8.5+ in CDH we use the time-out mechanism of Predictions so for every decision we make sure we only send one response to Adaptive (either a positive or a negative). In prior releases the recommended pattern was to send a negative at impression time always then trail it with an optional positive if the proposition was accepted/clicked. This results in double-counting the positives. For offer ranking this does not matter but the new approach is cleaner.
So what I am understanding from your above comment is even we have multiple impression (negative) and one positive (Click) it is fine from model performance point of view.
My next question is in that scenario we are feed both positive and negative response, ideally for a decision we should feed only one response either positive or negative? Is this fine to feed both positive and negative to model ? Does it will have any impact on propensity the model generates as positive and negative response is not mutually exclusive?