Question
Anthem Inc
US
Last activity: 22 Jan 2024 10:11 EST
Number of Threads to Number of Partition relation in Queue Processor
The default number of partitions after 8.7 for a queue process is 6. We can increase it using pyalterstreampartitions activity.
The question we have is, is there a 1:1 relation between the maximum of threads which can be spawned across cluster to the number of partitions?
Example: If i increase the number of partitions to 40, can i increase the number of threads across cluster for a Queue processor to 40? or the maximum threads allowed is always 20 for a queue processor?
Accepted Solution
Updated: 22 Jan 2024 10:11 EST
Bits in Glass
IN
Maximum number of threads across all nodes in a cluster for a QP is 20. That's the limit for a QP. Please check 8th point in the below url.
https://docs-previous.pega.com/system-administration/87/queue-p
And if you increase beyond 20, only 20 will process messages from a QP. And the threads should be increased on the CPU and heap. We need to ensure if Host server can handle the load before increasing the threads to 20. But maximum number of threads limit is 20 for a QP.
https://support.pega.com/question/increasing-thread-count-stream-node-and-custom-node
Pegasystems Inc.
CA
I believe it supports only up to 20 threads per cluster.
Anthem Inc
US
@RameshSangili Can you confirm on it? I didn't find any clear documentation on it. If you can provide any supporting documentation it will be of great help.
Updated: 8 Jan 2024 13:53 EST
Pegasystems Inc.
CA
Please find the links,
https://docs-previous.pega.com/decision-management/87/best-practices-stream-service-configuration
I hope this helps!
Bits in Glass
IN
@KarthickRajasingh in the older versions it was 20 partitions. Now in recent versions pega supports 6 partitions in a cluster. Please check 6th question in the below url.
https://docs.pega.com/bundle/platform/page/platform/background-processing/queue-processor-faq.html
Updated: 11 Jan 2024 9:59 EST
Tata Consultancy Services
IN
Could you please check the below articles? Very useful documents are related to Kafka and queue process documents
Note: My understanding is that from Pega Platform 8.7 onwards, it will create only 6 participations by default in 8.7 and before 8.7 versions it will create 20 participations by default.
If you want to increase or decrease the number of participants, you can run the below activity.
Activity Name: pxAlterStreamPartitions (This activity available in Pega 8.7 and above versions)
Parameters:
stream: Pass in Your QP name All in Capital Letters
partitions: Number of partitions, I have tested it, I gave the value 25 , It is allowed and Created the 25 partitions in Personal Edition 8.7
As above mentioned the Tool tip message, It is Same for the all versions, 8.6 and 8.7, The Value should be dynamic for every QP. Even The default value 6 partitions in the 8.7 Pega Platform, Still Tool tip message not updated and It is showing the 20.
https://onestoppega.com/background-processing/queue-processor-in-pega/
Could you please check the below articles? Very useful documents are related to Kafka and queue process documents
Note: My understanding is that from Pega Platform 8.7 onwards, it will create only 6 participations by default in 8.7 and before 8.7 versions it will create 20 participations by default.
If you want to increase or decrease the number of participants, you can run the below activity.
Activity Name: pxAlterStreamPartitions (This activity available in Pega 8.7 and above versions)
Parameters:
stream: Pass in Your QP name All in Capital Letters
partitions: Number of partitions, I have tested it, I gave the value 25 , It is allowed and Created the 25 partitions in Personal Edition 8.7
As above mentioned the Tool tip message, It is Same for the all versions, 8.6 and 8.7, The Value should be dynamic for every QP. Even The default value 6 partitions in the 8.7 Pega Platform, Still Tool tip message not updated and It is showing the 20.
https://onestoppega.com/background-processing/queue-processor-in-pega/
https://onestoppega.com/batch-processing/batch-data-flow-real-time-data-flow-in-pega/
https://docs-previous.pega.com/decision-management-reference-materials/kafka-streaming-service
https://docs-previous.pega.com/system-administration/87/queue-processor-faq
https://docs-previous.pega.com/decision-management/87/managing-data-flow-runs
Thanks,
Ashok
Anthem Inc
US
@Bhumireddy I completely understand and we did test from our end to make modifications on number of partitions but my question is related to number of threads. As explained by you, you can have 25 partitions but what is the maximum number of threads? is it 25 or 20 is the max?
Updated: 20 Jan 2024 0:00 EST
Tata Consultancy Services
IN
In my scanario, the maximum number of threads is 25, as I have increased the number of threads from 6 to 25. Still, if you want an increase, you can increase threads.
FYI, I have again increased 100 Threads,
PFA
Thanks,
Ashok
Accepted Solution
Updated: 22 Jan 2024 10:11 EST
Bits in Glass
IN
Maximum number of threads across all nodes in a cluster for a QP is 20. That's the limit for a QP. Please check 8th point in the below url.
https://docs-previous.pega.com/system-administration/87/queue-p
And if you increase beyond 20, only 20 will process messages from a QP. And the threads should be increased on the CPU and heap. We need to ensure if Host server can handle the load before increasing the threads to 20. But maximum number of threads limit is 20 for a QP.
https://support.pega.com/question/increasing-thread-count-stream-node-and-custom-node
Infosys Limited
IN
To test this try to execute this qp which would get execute all the requestors at once. I believe this might create the 40 threads (requestors). But the disadvantage of having the threads (requestors) it might eat up the resources.
To check the threads, open the dataflow associated with the QP and check how it behaves.