Question
Queue Processor - Kafka folder structure
Can you please suggest for below question regarding Queue Processor rule.
If we create queue processor rule on multi node cluster (ex: Background Processing = 5 Nodes ), on which node the Kafka topic (physical folder structure) will get created?
Will it setup on all nodes of cluster? Or only on specific one node? If answer is specific one node then where to specify the configuration for desired node.
Thanks in advance.
Queue processor runs in stream node via data flows and we usually have only one node defined as stream nodes , there is no restriction to make multiple nodes a steam node in a cluster , but suggestion is to use one node else we will see issues with data flow run.
In one node there can me multiple topics and each topic can have many partitions , KAFKA will create multiple threads for topic partition data processing .
@AbhilashBLR Hi Abilash, I have one some questions regarding the Kafka folder. Where exactly Pega creates the Kafka folder? If Pega creates the Kafka folder in Temp folder then what happens if we clean this folder when recycle the system? Also, can we configure the location for the Kafka folder to create in some other place?
Thanks,
Durga