@Saurabh Mittal  ⚠ This is a GenAI-powered tool. All generated answers require validation against the provided references

Kafka Offset Behavior

Regarding your first question about offset movement: Yes, the offset counter in Kafka is designed to move forward automatically after reading a message, regardless of whether the message was processed successfully or encountered an error. This is part of Kafka's core behavior - when a consumer commits offset N, the next consumption starts at N+1.

The offset management is typically handled at the consumer level, and by default, Kafka will continue to the next message in the sequence even if the current one causes processing errors. This explains the behavior you observed where clearing the queue allowed new messages to flow in - the consumer was likely stuck on processing (or attempting to process) the problematic message without properly advancing the offset.

Dead Letter Queue Implementation

For your second question about moving messages to a dead letter queue: While Pega data flows don't have a built-in dead letter queue mechanism specifically, this functionality can be implemented through custom handling. Here are your options:

1. Using Pega's Error Handling:
   • You can leverage the retry settings provided by the source shape in your data flow
   • By setting the dynamic system setting dataset/execute/save/statusFailOnError to true, you can configure the system to fail the activity if any record processing fails, which gives you visibility into failed records
2. Implementing Custom DLQ Handling:
   • You would need to create a custom error handling mechanism that:
     • Identifies failed messages
     • Routes them to a separate Kafka topic that acts as your dead letter queue
     • Maintains metadata about the original message and the error encountered
   • This typically involves implementing error handling logic within your data flow
3. Kafka Connect Approach:
   • If using Kafka Connect as part of your integration, it has built-in error handling options including DLQ functionality
   • This allows problematic messages to be automatically routed to a designated dead letter topic

The most common implementation pattern involves capturing the exception during message processing, logging relevant details about the failure, and then publishing the original message along with error information to a dedicated dead letter topic for later investigation or potential reprocessing.

References:

• Kafka Connect Error Handling and Dead Letter Queues
• Kafka Offset Behavior - Stack Overflow
• Error Handling via Dead Letter Queue in Apache Kafka
• Pega Kafka Data Set FAQ