Question
Capgemini
NO
Last activity: 2 Jan 2024 4:21 EST
Kafka implementation in Queue Processor and Job Scheduler
Team,
I have following questions on Queue Processor [QP] and Job scheduler[JS] based on how Kafka is being leveraged.
1. I can understand that Pega creates a Kafka topic for each QP we created. Each topic has 6 partitions[before v8.7 it was 20] and it is configurable. Can I know how the maximum thread allowed per cluster is still 20? How each thread is mapped internally to partitions? I thought that any threads more than partition size would be idle only.
Reference :
"Queues are multi-threading and shared across all nodes. Each queue processor can process messages across six partitions so that queue processor rules can support up to 20 separate processing threads simultaneously without conflict." https://academy.pega.com/topic/queue-processor-and-job-scheduler/v2 https://docs-previous.pega.com/decision-management/87/changing-default-number-partitions-topic#section_d2t_kjc_gsb
2. When using external kafka service, Do we have to create topic and map to queue processors's data set manually?
3. I see from the document that Job Scheduler uses Kafka. Can someone share how JS uses Kafka internally? How JS can able to run when stream nodes are unavailable if uses kafka ?
Team,
I have following questions on Queue Processor [QP] and Job scheduler[JS] based on how Kafka is being leveraged.
1. I can understand that Pega creates a Kafka topic for each QP we created. Each topic has 6 partitions[before v8.7 it was 20] and it is configurable. Can I know how the maximum thread allowed per cluster is still 20? How each thread is mapped internally to partitions? I thought that any threads more than partition size would be idle only.
Reference :
"Queues are multi-threading and shared across all nodes. Each queue processor can process messages across six partitions so that queue processor rules can support up to 20 separate processing threads simultaneously without conflict." https://academy.pega.com/topic/queue-processor-and-job-scheduler/v2 https://docs-previous.pega.com/decision-management/87/changing-default-number-partitions-topic#section_d2t_kjc_gsb
2. When using external kafka service, Do we have to create topic and map to queue processors's data set manually?
3. I see from the document that Job Scheduler uses Kafka. Can someone share how JS uses Kafka internally? How JS can able to run when stream nodes are unavailable if uses kafka ?
Thanks !
Maantic Inc
US
1. You can try updating partition for one queue processor and check.
use activity - pxAlterStreamPartitions to update partition - https://docs-previous.pega.com/decision-management/87/sizing-queue-processors-and-stream-data-sets-individually-changing-number-partitions-topic?
2. Queue processor creation automatically take care creating topic and all the related work in external Kafka.
3. We moved to external Kafka, stopped all the stream nodes, I don't see any issue in job schedulers.
Capgemini
NO
@JohnPaulRaja,C Thanks for the reply, John. Please find the followup questions inline.
1. Sure, Can you share more info on how to check the partition mappings to node threads ?
2. Do we have to provide elevated access to Pega Application to external Kafka to create topic as well via application? Is this followed for PROD env as well ?
3. Got it. May I know how JS is working in this scenario if Kafka is used for processing ?