Processing large amount of data using Queue Processor
Hi Everyone,
I have to process large amounts of data (approx 5 lakhs rows) from the data type.
Currently, I am processing using 1 dedicated Queue Processor but it is taking a long time to process. so introduced another QP of the same configuration,
Now I am processing with 2 QPs, 1 for the pre-processing (validation and case creation), and 2nd QP for executing the flow. I have worked on it and seen that time is reduced a lot.
Is it a good idea to have 2 queue processors, or Is there any better way to address this?
Thanks for taking the time to address the above questions
Accepted Solution
Updated: 3 Mar 2023 1:03 EST
Bits in Glass
IN
@AAV7601 You can try to improve performance by increasing threads/dedicated nodes which increases the vertical/horizonal scaling capability. And for your question yes it is fine to have 2 queue processors. Please refer this link for reference . In the Queue processor explanation it says that Queues are multi-threading and shared across all nodes. Each queue processor can process messages across six partitions so that queue processor rules can support up to 20 separate processing threads simultaneously without conflict. Leveraging multiple queue processors on separate nodes to process the items in a queue can also improve throughput.https://academy.pega.com/topic/queue-processor-and-job-scheduler/v2#:~:text=Each%20queue%20processor%20can%20process,queue%20can%20also%20improve%20throughput.
You can check the suggestion mentioned in the bottom of this below reference
https://support.pega.com/question/queue-processor-how-improve-performance
Maantic Inc
IN
@Anoop Krishna Thank you so much for your valuable comments.
In my scenario, a case will pass through two different Queue Processors. I think this is right as mentioned in the PDN module.
Lventur
IN
Increasing the QP count alone will not be enough..
You need to work on infrastructure/sizing also. Because system trying to process 5million records which is huge, there is possibility of getting below issues.
- If QP is running continuously under particular node more than the threshold limit, then entire node will be down.
- CPU memory utilisation will be too high
- There is possibility to get heap memory out issue also.
Could you please let us know what is the source of 5million records, that would help us to give some additional design inputs..
Maantic Inc
IN
@Gunasekaran Baskaran this could be the worst case that there will be 5 lakhs cases (not 5 million), normally it will be in thousands.
Thanks for the reply