Kafka dataset in a real time dataflow in Pega 8.3
Hi All,
I have configured kafka connection and created kafka datasets for topics in my broker. Everything looks fine till now. I am working on two scenarios.
Scenario 1: Pega as a publisher: It is working as expected. I am able to push the data to a particular topic in pega using a data flow. No issues till now.
Scenario 2: Pega as a subscriber: I created a real time data flow run and initiated it. Now I am calling an activity at the destination of the data flow. I used a log step to check whether the message is received or not. Now, I started sending messages into the Kafka topic which is configured. It is streaming and the log is coming as expected.
And now, I am sending JSON text into the topic. How do I read it from the data flow destination activity? What is the page context of the input messages from the topic? I think I am missing something.
Please let me know. Thanks in advance
CBA
AU
Hi Nizam,
You can use DataSet-Execute method in activity to read the messages from Kafka topic using browse method and the results(pxResults) can be accessed from code-pega-list class page.
Decision Consultant Solutions
GB
Create a data flow to read the messages from the dataset as a source and be the activity as a destination.
and use the data transform, composition or convert shapes according to your needs to convert the data or map to properties.
Reference links:
https://collaborate.pega.com/question/how-map-json-message-clipboard-using-data-transform
Note: I see your use-case as realtime processing and you want to process each message from kakfa and process it in the activity.
The Data-Set Execute Browse is used to read a set of messages from Kafka dataset and not much useful in case of real-time data flow processing.
Rabobank
NL
Hi-
I have the similar use case to read messages from topic. If i run data set manually. Results are coming fine. But if i run it using data flow, there are no record processed ( 0 record has been processed). Also there is no logs. I am not sure why Dataflow is not picking the records. Could some one help on this?
Accenture
US
Hi Nizam,
I am facing the same problem as you described. Have you found the solution? I could create a message in Kafka's topic, but do not know which property/page we should map to. Also, the same problem, while retrieving the message from Kafka.
Thanks
Abdul
Adqura
TR
Hi,
I have a similar one, and it's working just fine.
Do you see the number of incoming counts on Real-Time processing landing page?
If your real-time data flow is in e.g. MyCRM-Data-Response class then keep your activity in the same class as well. This is the class that has represents incoming JSON payload properties. Since you're in the same class, you should be able to access the required properties right away.
Regards,