Question
Rulesware LLC
SV
Last activity: 29 Apr 2025 14:24 EDT
Where does the data gets stored after the source is executed in a Data Flow?
I have been trying to explore the capabilities of data flows, and I have the following use case:
I'm working with a Real-Time Processing Data Flow, where the source is a Kafka Data Set stored in JSON format. The Data Flow processes records as expected.
This is the data flow:
I'm trying to Log the data from the "TestDataSet" in the data transform, the problem is, I don't know where the data extracted from the source is stored, or how to access it.
Does anyone know how Pega loads the data after the source runs?
I tried accessing the "OperationResults" page (which is generated when you run the data set alone), but that didn’t work.
Accepted Solution
Updated: 29 Apr 2025 14:24 EDT
Rulesware LLC
SV
@SalvadorC94 To answer my own question and in case someone has the same question in the future, here is the solution.
Since it is a real-time data flow you are not able to fully trace the whole process. So, you depend on Logs inside a Data Transform or activity that you put on your Data Flow Rule.
To be able to access your data and see if everything is working as expected using Logs here are the steps I performed:
To see the data structure, you can force an error in your Data transform or Activity (performing a division by zero usually does the trick) so you can see the failed records in the Real-time Data Flow Process Landing Page. Then you create the properties as needed to be able to access the data and do the Logs.
More details on the attachment: