@JohnPaulRaja,C

Answer of 1st Question:-

When you start the dataflow, there will be 20 consumers created - one for each partition in the topic. Since you have configured your data set to have a partition key of pyid, each consumer will read messages only from the partition corresponding to the pyid of the message. Therefore, there will be a maximum of 20 consumers in your setup.

Answer of 2nd Question:-

To ensure that messages are consumed in the correct order, you need to ensure that messages with the same pyid are assigned to the same partition. This can be done by using a partitioner that assigns messages to partitions based on their pyid. You can either use a built-in partitioner in Kafka or write a custom partitioner that implements the Partitioner interface.

Once you have ensured that messages with the same pyid are assigned to the same partition, Kafka will guarantee that the messages will be consumed in the order they were produced within each partition. Therefore, if the producer publishes two messages with the same pyid, they will be assigned to the same partition, and the consumer will consume them in the same order in which they were produced. However, if the producer publishes messages with different pyids, Kafka does not guarantee the order in which they will be consumed, even if they were produced in a specific order.