To gain clarification on the section 'How do I increase the performance of my queue processor rules?' in our latest  documentation, could I ask that you submit a documentation feedback by clicking on the 'Feedback' (speechbubble) icon on the top right of the Queue processor FAQ document?

I can see no difference in the description between the Pega 8.7 Queue processor FAQ and the Pega '23 Queue processor FAQ .

However, the difference is mentioned in our Pega Academy documentation:

Queue processors and job schedulers  (from the Advanced background processing v4 module for Platform 8.8)

The reference to  '20 separate processing threads per cluster' is omitted in later mission: 

Queue processors and job schedulers (from the Advanced background processing v5 module for Platform '23)

@Bhumireddy starting from 8.7 Pega platform defaults the QP partition count to 6 (earlier 20) and provided an extension to alter appropriate QPs.

On Pega 8.6.4+ versions, clients have the option to alter the partitions only for specific topics through the activity pxAlterStreamPartitions - in case this approach is used, the corresponding topic needs to be manually updated to match with the new partition count (Kafka only supports increasing partitions for an existing topic - decreasing the number of partitions will require re-creation of the topic).

Users can use the OOTB pxAlterStreamPartitions activity to increase the partition count for specific QP's that require high throughput and parallel processing.

Sizing queue processors and Stream data sets individually by changing the number of partitions of a topic

Sizing queue processors and stream Data Sets individually by changing the number of partitions of a topic

Also note that pxAlterStreamPartitions reset the stream partitions count to default 6 during the case if Kafka data directory is also cleared or removed during restarts. In such cases, we have to redo the step.

To answer your question about activity 'pxAlterStreamPartitions' : You need to pass the QP name and number of partitions are 6. Then verify the QP dataflow if the partitions are reduced to 6. If not, please stop the QP from admin studio and run 'pxRestartByID' by passing the QP name. Please check the checkbox 'Allow dataloss' while running this activity.

-----------------------------------  Details ----------------------------

When using external Stream, you actually need to pass the full Kafka topic name for the Queue Processor in question which you’re attempting to alter the partitions count.

With the addition of external stream support, it was included a new DSS/Prconfig to customize the prefix and/or suffix for all Pega related topics, so you have the option to point multiple Pega instances to the same external Kafka cluster using unique topic names. This customization possibility was added to avoid multiple Pega instances attempting to use the same topic names. Following is the DSS/Prconfig in question:

• Dynamic System Setting:

Owning Ruleset: Pega-Engine Setting Purpose: prconfig/services/stream/name/pattern/default Value: prefix-{stream.name}-suffix

• Prconfig:

<env name="services/stream/name/pattern" value="prefix-{stream.name}-suffix"/>

Please note that the variable {stream.name} should be always present in the pxAlterStreamPartitions activity value as it will get replaced by the kafka topic name during runtime.

------------------------------------------------

1. Older Pega versions:

@Anoop Krishna clarified in this thread  Number of Threads to Number of Partition relation in Queue Processor that  in the older Pega versions it was 20 partitions. :

Maximum number of threads across all nodes in a cluster for a QP is 20. That's the limit for a QP. Please check 8th point in the below url.

https://docs-previous.pega.com/system-administration/87/queue-p

And if you increase beyond 20, only 20 will process messages from a QP. And the threads should be increased on the CPU and heap. We need to ensure if Host server can handle the load before increasing the threads to 20. But maximum number of threads limit is 20 for a QP. 

https://support.pega.com/question/increasing-thread-count-stream-node-and-custom-node

2. Latest Pega versions:

@Bhumireddy specified that now in recent versions pega supports 6 partitions in a cluster. (see 6th question below):

https://docs.pega.com/bundle/platform/page/platform/background-processing/queue-processor-faq.html

--------------------------------------

Clarification requested in this PSC post:

@nairv1  @REDDR3  can someone comment about the above statements?

Hi @nairv1

I totally understand, from Pega 8.7 On wards, 6 partitions are created by default  for One Queue Processor.

What is the maximum number, I mean Upto 8.6 we have 20 partitions, From 8.7 I can't see the maximum number of partitions.

We have a activity to increase the number of partitions.(Activity Name: AlterStreamPartitions).

And also, this tooltip message is still showing the 20 numbers in Enterprise Edition 8.7.6 ( and Personal Edition 8.7) if the 6 partitions are created by default for the One Queue Processor, if we give 20 number, how will it process?

Note: It will process only 6, because 6 partitions are created by default.

I can see It showing the 6 partitions in Community Edition, Nevertheless, It should be always dynamic right I mean it should be fetched value from the Data Flow rule right. (If we increased number of participants, then It shows the different value in Data flow rule and tooltip showing the different value)

Could you please address this? I am getting confused. And what is the maximum number of partitions can you recommend?

Let's suppose we have 100 million and more records there, and we have to process them.

Thanks,

Ashok