Question
Merkle
GB
Last activity: 12 May 2021 12:15 EDT
Process events via Kafka realtime processing
When I restart the ProcessEvents dataflow, we can successfully read the events from external kafka. We’ve configured the read options for the dataset as ‘Read existing and new records’, which means, when we restart, its reading the existing events and the same we could trace. Just want to understand, since how many days old the existing events gets picked up? Because, there were more than 2000+ events existing in Kafka and only 750+ events processed after restart.
Merkle
GB
I believe, Pega is processing 2 weeks old events, but would like to know if any such configuration in Pega or is it something Kafka might be allowing to consume 2 weeks old events?
Pegasystems Inc.
IN
Hello @Venkataiah Kandula
Is this post related to Kafka Dataset or Stream Dataset?
Merkle
GB
@PoojaGadige it is related to Kafka dataset, configured in a Dataflow, and the dataflow configured under Real-time processing.
Pegasystems Inc.
NL
"Read existing and new records" should read all the records , so in your case all the 2000 records. The option, "Only read new records" will read records sent once the real time data flow has started.
Merkle
GB
@kaman , thanks for the response. So, the 2000 records are in a span of a month. and we could see only 750+ records got processed. What do you think the cause for not processing the whole records? as I mentioned, the 750+ records are in a span of 2 weeks.