Question
DHL
IN
Last activity: 25 Jun 2019 23:58 EDT
Kafka Data Stream Data Flow Consumes duplicate data when real-time processing dataflow restarts
Hi ,
We are consuming data from kafka topic. Realtime dataflow is configure where kafka stream data set is configured.
While testing i could see when i stop the realtime dataflow and restart it,its processing all the records which its processed before stop.
EX:
kafka topic has 80 records and real time dataflow read all 80 records(for testing logging the data) and now Data flow is stopped.
When i restart the data flow, again it is reading the 80 records which was already processed.
Is there any setup/configuration that should needs to be done at pega so that it will not read data again.
Thanks.
Pegasystems Inc.
NL
Hi Sivajyothi, is the data flow source configured to read existing and new records, or only new records?
DHL
IN
Hi dacoi,
Thanks for your response.
data flow is configured to read existing and new records.
If we configure it as only new records,its working fine by reading records only after the dataflow configuration.
the problem here is , if we configure read only new records and by what ever reason dataflow got stopped in the middle and at this point kafka topic has 500 records which are not processed by pega. When i restart dataflow, as its read only new records we are going to loose 500 records which are already in kafka topic.For this reason we choose for read Existing and new records,but with this configuration whenever data flow is restarted its processing all existing records again.
Pega platform - 8.1.4
In 7.3 we could see read from begining option in kafka stream data set which is missing in 8.1.4