There is no configuration to configure Kafka-consumer-group or group.id in the Pega consumers which is crucial when consuming messages from Kafka to ensure reliable and scalable message processing and not setting a consumer group can lead to inefficiencies, inconsistencies in message processing, potential message loss, and difficulties in managing and monitoring consumer behavior.

To brief on what is Consumer group: It a feature or logical group of consumers which will have unique group ID and helps to determine which consumers belong to which group or who is the consumer for the particular Topic.

Current challenges faced without consumer group:

• Our Kafka ASG is finding difficult to find which is the consumer that is consuming a particular Topic.
• There have been multiple consumer groups that got assigned with the name consumer-****  in Kafka environments by default and we find difficult to set monitoring alerts when a message is stuck at Kafka queue.
• During the failure scenarios lets consider when Pega restarts their consumer jobs and every restart, application will set a new group-id as there is no consumer group setting, due to this task there may be loss of data or duplicates.

Data Duplicate scenario : Lets consider 10 Messages are present in the Topic i.e. offset 0 to 9 and when pega job fails at 4th Message ,offset committed was is 0,1,2 and when the job restarts and you have set the consumer to consume from beginning of the Topic it will again start from the first to consume 0,1,2.

Data Loss scenario : Lets consider 10 Messages are present in the Topic i.e. offset 0 to 9 and when pega job fails at 4th Message ,offset committed was is 0,1,2 and when the job restarts and you have set the consumer to consume from latest messages of the Topic it will start from the 9th offset where 4th  to 8th is lost.

Currently, to mitigate this concern, we have resorted by granting a wildcard (*) ACL, allowing Pega to pick any consumer-id by default. However, this workaround poses challenges for us in understanding and monitoring which consumer group Pega is utilizing. Therefore, it has become imperative for us to address this issue promptly.

 Hence, Requesting Pega Teams to provide insights regarding the concern and raise this with pega support if they can provide any solution or how sooner this can be implemented.