Question
Areteans Technology Solutions
IN
Last activity: 13 Oct 2017 3:13 EDT
Pega marketing campaign launch benchmark
Hello,
We are in urgent need to provide benchmark to our client regarding average time it takes Pega Marketing to launch a campaign i.e when an campaign is executed in production how much average time it takes to execute the segment, then execute the strategy , execute the offer flow and eventually sending the sms.
Let’s take 3 types : simple campaign, medium complexity campaign and high complexity campaign with 1-5 offers for each.
Does pega provides any such benchmark?, if yes kindly share at the earliest as customer is demanding for it.
Thanks in advance
Regards
Abhi
Pegasystems Inc.
GB
Hi, there's no benchmark data for this kind of activity as its very dependent on multiple dimensions of the data we're working on.
As an indication, segment processing is a SQL based operation, the time it takes to populate a segment is very much determined by the query used to satisfy the segment criteria, the indexes available and the volume of data being inspected. If it's a visual segment, there'll be additional queries fired to count the rows in the various parts of the Venn diagram, these all contribute to the segment processing time.
Strategy execution is again completely dependent on what it's doing but in batch we urge customers to measure their strategy execution time and we'd expect it to take no more than70 milliseconds per customer, the higher it is the lower the throughput. Dataflow execution is highly scalable and so you would be able to achieve the throughput rate you need by adding resources either horizontally or vertically, but the higher the cost per customer the lower the throughput rate will be.
Offerflow processing, as distinct from treatment delivery, is pretty linear and incrementally grows as more shapes are in the Offerflow. It's also wrapped up now in the main dataflow execution as of 7.22 so, the statement above applies.
Hi, there's no benchmark data for this kind of activity as its very dependent on multiple dimensions of the data we're working on.
As an indication, segment processing is a SQL based operation, the time it takes to populate a segment is very much determined by the query used to satisfy the segment criteria, the indexes available and the volume of data being inspected. If it's a visual segment, there'll be additional queries fired to count the rows in the various parts of the Venn diagram, these all contribute to the segment processing time.
Strategy execution is again completely dependent on what it's doing but in batch we urge customers to measure their strategy execution time and we'd expect it to take no more than70 milliseconds per customer, the higher it is the lower the throughput. Dataflow execution is highly scalable and so you would be able to achieve the throughput rate you need by adding resources either horizontally or vertically, but the higher the cost per customer the lower the throughput rate will be.
Offerflow processing, as distinct from treatment delivery, is pretty linear and incrementally grows as more shapes are in the Offerflow. It's also wrapped up now in the main dataflow execution as of 7.22 so, the statement above applies.
Treatment delivery (sending SMSs or Emails) is completely dependent on the latency when communicating with the target delivery platform ( ESP for emails and SMSCs for SMS ), it's not uncommon for emails to cost ~ 1/2 second to to an ESP, high degrees of parallelism in the Pega layer can offset these and we've had implementations get up to 300 emails/second
Per the opening statement, these are just data points and anecdotal information, there’s no published numbers but I hope the above helps you frame your discussion with your customer.
Areteans Technology Solutions
IN
Thanks Nigel.
I completely agree that Campaign throughput depends on multiple dimension. But what we were expecting is that pega before releasing Pega marketing as a product must have done some load testing and came up with a benchmark e.g if we use a specified infrastructure, server configuration, x millions customer base and segment having this many joins (on an avg ) , strategy having this many shapes, given latency with target delivery platform and parallelism implemented in pega we were getting this throughput and performance figures etc..
We were expecting something on this line.. so that using this as a base figure we can extrapolate expected performance figure for our project.
Do you think we can have anything like this?
Regards
Abhi
Areteans Technology Solutions
IN
Hello Nigel,
Can you provide some figures for SMSCs for SMS i.e time taken to send SMS.
In our last campaign run the segment and strategy execution was at par with above mentioned figures, however time taken to send SMS to 1.4 million customers is taking 11 hours.
Regards
Abhi
Pegasystems Inc.
GB
Hi, I don't have figures, but the SMS agent (like the email agent ) is able to utilize multiple threads to achieve high throughput. The figure you've quoted ( 11 hrs to do 1.4 million customers ) is very slow.
Can you elaborate on how you're sending SMSs and what the time of transmission for one message is to the SMSC is?
The email and sms agent both share the same approach of pulling messages from the correspondence table and processing them in batches with multiple child requestors, if you put logging on the agent activity, you'll get details of the parallelism it's processing.
In answer to your other question, we don't have public data on the performance, In my experience, it's never the same from customer to customer.
Areteans Technology Solutions
IN
We are using outbound SMS listener to connect to our client SMSC services, we have taken some PAL readings (see attached) while sending test Messages from the SMS treatment in our PROD environment. On an average is taking more than half a second to send a single SMS.
Regards
Abhi
Pegasystems Inc.
GB
I'm not familier with PAL readings but given each request is taking > 0.5 seconds, then yes, you'll need to increase the number of threads (requestors ) being used to send the SMS stream. That's done on the Decisioning -> Infrastructure -> Simulation -> topology landing page. The threadcount is limited by the maximum child requestors prconfig.xml parameter which will need to be increased. I've seen customers get the thread count into the low hundreds to achieve the throughput they need if the latency of the target service is slow. However, before heading down this path, verify that the SMS agent is indeed spawning off multiple child requestors ( it should match what your threadcount on the topology landing page is ) by setting the logging level to Info for the main activity.
Areteans Technology Solutions
IN
Thanks Nigel for the suggestion. Even though we are quite sure that SMS agent is spawning off multiple child requestors by see the below code snippet from SendSMSInBulk agent activity , but we will double check anyway.
However an interesting thing to note as per below code snippet if we try to increase the number of connections to more than 10 then pega marketing 7.12 uses only 5, so which means max limit is 10, so not sure how to increase it to hundreds will achieve better throughput, please clarify.
Additionally as per the agent activity code, it spwans multiple threads in a single node only, does pega marketing 7.22 supports multinode processing, that in my opinion will better the througput.
Thanks Nigel for the suggestion. Even though we are quite sure that SMS agent is spawning off multiple child requestors by see the below code snippet from SendSMSInBulk agent activity , but we will double check anyway.
However an interesting thing to note as per below code snippet if we try to increase the number of connections to more than 10 then pega marketing 7.12 uses only 5, so which means max limit is 10, so not sure how to increase it to hundreds will achieve better throughput, please clarify.
Additionally as per the agent activity code, it spwans multiple threads in a single node only, does pega marketing 7.22 supports multinode processing, that in my opinion will better the througput.