Kafka Message Consumption by Multiple Pega Deployment
Hello,
We have a need to publish message to a kafka topic (through queue for processing method) and the same has to be consumed (through QP )by another workload/pega deployment.
Context :: These workloads will share a common kafka cluster(Externalized)
Also, if multiple workload(s)/pega deployments have to consume the same message, then is it recommended to create QP Rules in the respective workloads/systems that are sharing the same external kafka cluster for processing?
Also, if we are reusing the same cluster for multiple pega deployments, I think the below setting has to be common across the pega deployments sharing the kafka cluster correct? Please confirm
<env name="services/stream/name/pattern" value="pega-dev-{stream.name}"/>
Pegasystems Inc.
NL
Manheim
US
@malaa1 . Thanks. Because your recommendation is to not use Q-F-P method for the above use case, what other options we have to post a message to a kafka topic (if we aren't using Q-F-P Method)? I believe we shall use Dataset-Execute insert option?
Let's say we are creating a data-set(kafka), should we be creating the topic ahead in the external kafka cluster and use the select from list under the topic section or should we use create new topic? My question is if we destroy and bring up the kafka cluster fresh, how can we ensure that the topic names in use are auto-created when the kafka cluster is brought up if we use select from list option? I believe it would be resilient to use the "Create new topic" so that each time the kafka cluster is destroyed and restarted, it will create a new topic and work as expected. Can you confirm?
Secondly, our goal is to have more than 1 consumer read from the dataset. Application/Pega deployment A queues the message, Pega Deployment B and Pega Deployment C needs to process the message. I know that we shall use data flow to process messages from the dataset, however can you confirm if we can have 2 data flows(independent running on respective/distinct workloads) process/read from the same dataset? Is this recommended? If a message is inserted into the kafka topic, will both data flows catch and process the same?
@malaa1 . Thanks. Because your recommendation is to not use Q-F-P method for the above use case, what other options we have to post a message to a kafka topic (if we aren't using Q-F-P Method)? I believe we shall use Dataset-Execute insert option?
Let's say we are creating a data-set(kafka), should we be creating the topic ahead in the external kafka cluster and use the select from list under the topic section or should we use create new topic? My question is if we destroy and bring up the kafka cluster fresh, how can we ensure that the topic names in use are auto-created when the kafka cluster is brought up if we use select from list option? I believe it would be resilient to use the "Create new topic" so that each time the kafka cluster is destroyed and restarted, it will create a new topic and work as expected. Can you confirm?
Secondly, our goal is to have more than 1 consumer read from the dataset. Application/Pega deployment A queues the message, Pega Deployment B and Pega Deployment C needs to process the message. I know that we shall use data flow to process messages from the dataset, however can you confirm if we can have 2 data flows(independent running on respective/distinct workloads) process/read from the same dataset? Is this recommended? If a message is inserted into the kafka topic, will both data flows catch and process the same?
Lastly, I know that we have the ability to start data flows manually through the system landing page, however how can we ensure that the configured data flow automatically starts and runs each time the batch/processing cluster is restarted?
Manheim
US
@malaa1 Also, I am curious to know which dataset Pega uses under the covers when we use Q-F-P method? This is a totally different question/topic, however I would like to know what happens under the covers? Imagine that we are running stream/kafka as an external service, when we call Q-F-P, how is the kafka topic getting created? Is this based on the QP rule name? and does it create a topic using "Pega" as the prefix by default?
Is below the setting to employ for overriding the same?
<env name="services/stream/name/pattern" value="pega-dev-{stream.name}"/>
Hi @SethuS88, Thanks for reaching out. Ideally queue processors are built to handle the workload of a platform (deployment) in the background to process the messages.
Coming to the use-case, Instead of queue processors either stream dataset or Kafka dataset are better suited for this scenario. I would recommend exploring this option.
Also, we don't recommend 2 different pega installations share the same value for below property. This property has to be unique across even though they are using /sharing same kafka cluster.