Question
Capgemini
US
Last activity: 5 Jun 2023 5:17 EDT
How to read the real time message from a Kafka Topic
We are working on following use case. If anyone implemented pls let me know.
- External system posts real time message to Kafka topic.
- Requirement is to read the data from Kafka time (new messages) and load the data to Pega table.
As per my initial analysis we need to create a Dataset(Type= Kafka) and Realtime data flow. Select Source as Dataset and Database table as destination.
With this option if we run the data set manually it would work, however I am looking for message reading should be automated. I assume Kafka Dataset won't read the real time message.
Cognizant Technology Solutions
IN
@SrinivasS1572, I think you can configure Job Scheduler and in Job Scheduler activity, you can use Data-Set-Execute method to run the data set.
Hope it should work for you!
Best Regards,
Vijendra HD
Capgemini
NO
Hi @SrinivasS1572,
Yes, your analysis is correct. To read message from Kafka, we need Dataset rule to configure kafka topic to be read and Dataflow rule to save to run activity or to save to database table
To listen messages in Realtime, you have to configure to run the dataflow as realtime in the system.
You can do it from CONFIGURE -> DECISIONING -> DECISION -> DATA FLOWS ->REALTIME DATAFLOWS -> New. There, you have to configure options like NodeType to run on and Number of threads for parallel processing.
Ensure you have added appropriate "Read Options" in Dataset rule.
Let me know if you need more information.