Question
Royal Bank of Canada
CA
Last activity: 8 Oct 2021 8:56 EDT
Queue processor on data objects
Hi,
I have a requirement to get data class records like 1000 and iterate them, call a service on each record and save each data instance into another data table with the response from service. As we need to process 20,000/- records per day, that means 20,000 service calls I am thinking to utilize queue processors so they can share the work load and do not impact the system performance. But my question is can we call a queue processor on a data class page and queue the data objects? As, queue processors are same like Standard agents which we use on work objects I am not sure whether we can utilize queue processors for data objects.
Right now we are using Job scheduler for this requirement but each hour getting 1000 records and processing them for 20 hours per day may impact the system performance so, thinking about an alternative. Could someone please suggest me a better approach?
Thanks in advance.
Accepted Solution
Updated: 8 Oct 2021 8:56 EDT
Ernst & Young LLP
MT
I think both the queue processor and standard agent can handle any class instance, be it a data class or work class that is saved to a DB table (Queue processor can even handle page snapshot).
Using a Queue processor seems to be the best option in your scenario, IMO.
Couple of suggestions -
- Remember to adjust the 'Number of threads per node' settings on your Queue processor for optimized performance. Normally, it's calculated in this way : 20 / No. of Nodes where this queue processor will be enabled.
- Given that for each instance you queue, an external system/service will be contacted via Connector, keep the network bandwidth requirement in mind.
- Queue processor can fork multiple parallel threads on the same node to process multiple queued instances in parallel. So, check with the external service if they can handle at max 20 (in worst case scenario) parallel connection to their service (The same limitation we face when we setup a service at PEGA end i.e. service requestor pooling).
Hope it helps.
Royal Bank of Canada
CA
Hi, Thank you very much for your valuable suggestions which helps me a lot. I will try this solution and follow the suggestions and hope it works :)