Question
Maantic Inc
US
Last activity: 23 Feb 2023 12:36 EST
kafka guarantee message ordering
Hi,
Producer having partition key pyid and message are published to topic
Topic having 20 partition
Topic consumer is pega application - real time dataflow
Data set configured to have partition key pyid
Dataflow run configured to 5 threads per node in Background nodes (total bg nodes 10)
Question 1:
When we start the dataflow, how many consumers are created?
Question 2:
How to make sure messages are consumed in correct order (kafka guarantee message ordering)
if producer publishing two message with same pyid, consumer should consume in same order
Bits In Glass
IN
Answer of 1st Question:-
When you start the dataflow, there will be 20 consumers created - one for each partition in the topic. Since you have configured your data set to have a partition key of pyid, each consumer will read messages only from the partition corresponding to the pyid of the message. Therefore, there will be a maximum of 20 consumers in your setup.
Answer of 2nd Question:-
To ensure that messages are consumed in the correct order, you need to ensure that messages with the same pyid are assigned to the same partition. This can be done by using a partitioner that assigns messages to partitions based on their pyid. You can either use a built-in partitioner in Kafka or write a custom partitioner that implements the Partitioner interface.
Once you have ensured that messages with the same pyid are assigned to the same partition, Kafka will guarantee that the messages will be consumed in the order they were produced within each partition. Therefore, if the producer publishes two messages with the same pyid, they will be assigned to the same partition, and the consumer will consume them in the same order in which they were produced. However, if the producer publishes messages with different pyids, Kafka does not guarantee the order in which they will be consumed, even if they were produced in a specific order.