Hi ,

We are observing an unusual behavior of Kafka stream data set every day at particular time(between 23:00 - 00:00) where it is reading all the records of kafka topic again.

We are connecting to exterenalkafka system and the Kafka configirations are done as below at Pega:

-  configured Kafka Instance to connect to external Kafka server

- configured Kafka stream data set 

- configured Real time data flow with Kafka stream data set as source and with 'only read new records' option

Example scenario: 

- Kafka Topic got 100 records in a particular day and pega reads them as and when they are available in Kafka topic.

- At end of the day(between 23:00 - 00:00) , again reading all those 100 records.

Pega Version : 8.1.7

Server : WAS Liberty.

Please provide any inputs .Thanks in advance.