Question
American Express
IN
Last activity: 29 Dec 2023 21:21 EST
Queue Processor- Partitions Understanding
Hi Team,
As per my understanding, whenever we create a Queue Processor, system will create 6 partitions (recent version) for that particular Queue Processor.
Consider I have a single node and single thread.
Now, I'm calling my QP activity for 10 times parallely. So it will create 6 messages in 6 partitions at starting, what will happen to remaining 4 messages?
When it gets added to Partitions and will it be stored anywhere before pushing it to any of the Partitions?
Also requesting to give relation on the Node type and Number of threads mentioned in the Queue Processor against the Stream Node (Kafka processing)?
Please correct me if I'm wrong.
Maantic Inc
US
American Express
IN
@JohnPaulRaja,C Hi John, understood clearly Thanks for your input.
Putting one more question below.
As per my understanding once message is sent to partition, then real time data flow will process the message and it executes the QP activity.
Now question is, we are mentioning number of threads in Queue Processor Activity rule. Is this denotes both the real data time flow threads and the activity execution threads or which thread it denotes?
Also we are giving node type in QP. That means my real time data flow will process the message on stream node and it will move it to QP node type to execute my activty?
Thanks, Vignesh.
Maantic Inc
US
1. QP and activity together it is one thread
2.whatever node type picked in QP, it runs on the node.
example : if you configured background node in queue processor, QP uses background node for execution.
Tekclan Software Solutions
IN
Hi @Vigneshk2107,
As per below article, at a time 20 partitions are available so able to process 20 partitions. Please refer the article:Queue processors and job schedulers | Pega Academy
Thanks.
American Express
IN
@DhanasekarC0202 Hi Dhana, If I'm not wrong as per recent version 8.7, it was changed to 6. i.e Whenever a QP created, PEGA will create 6 partitions.
Please correct me if I'm wrong.
Updated: 29 Dec 2023 10:58 EST
Tekclan Software Solutions
IN
Hi @Vigneshk2107,
we can check the partitions size in \tomcat\kafka-x.x\config server.properties file the variable name is for partition count EX: num.partitions=20 based on this count it will work.
Thanks
@Vigneshk2107
For your configuration 6 partition and one consumer(thread) is created to process the message.
if you are using queue for processing with queue immediately - all the ten messages will pushed to Kafka topic,
pega default using round robin algorithm to queue the messages in topic, so all your four message in some partition. (surf more about partition round robin algorithm to understand)