Question
ING Nederland
NL
Last activity: 19 Apr 2024 4:02 EDT
Retrieve the JSON for a Kafka data set.
Hi everyone,
We have a data flow which is sourced by a Kafka data set. The data set uses Avro data format and we operate as subscribers to this Kafka event.
Our requirement is to extract and store the whole json which we have received before using the Convert shape. So, let's say we want to store the received json in a pyNote property and use it for further processing. Can someone please help on how this can be achieved?
Thanking you in advance!
Regards,
Saurabh
Accepted Solution
Updated: 19 Apr 2024 4:02 EDT
ING Nederland
NL
We have an activity which is called at the end of the dataflow (data set -> convert shape -> activity).
In this activity we copy the primary page(which is the de-serialized page) to a temp page & then using a JSON DT with auto-map feature we again serialize the temp page to get the result in param.jsonData.
P.S. In the settings tab of the JSON DT exclude the OOTB properties such as pxObjClass etc.
Regards,
Saurabh.
Maantic Technologies Pvt Ltd.
IN
Hi @SaurabhG7134,
You can use the "Use data transform" option under "Message values" option. Create a data transform which can get the JSON from the data set and set it to a property on the primary page of the data flow.
ING Nederland
NL
Hello @SrijitaB, that option is to map the schema to the Kafka topic wherein we upload the json format. I am not sure how can we capture the response json from here.. Can you please share some screenshots to explain further.
Regards,
Saurabh.
Updated: 9 Apr 2024 7:50 EDT
Maantic Technologies Pvt Ltd.
IN
ING Nederland
NL
@SrijitaB This is for the mapping from JSON to clipboard :) I need to get the full JSON similar to how the param.jsonData from the "SERIALIZE" DT behaves.
Maantic Technologies Pvt Ltd.
IN
Yes I am very well aware of the fact that this option is for mapping the JSON to clipboard :-)
May be I will have to describe the solution in detail for you to understand.
- Create a clipboard type data transform.
- In the data transform have one step and set <.any property name> = Param.jsonData
- Call this data transform from Message Values -- >> Use Data Transform option
- Run the Data Set / Refer the Data Set in your data flow to get the entire json in your clipboard primary page in the property you referred in step #2
Note : You will not get this clipboard Data Transform in the "Use Data transform option" as it expect a JSON type. But you will very well be able to save and execute the data set.
ING Nederland
NL
@SrijitaB Thank you for the explanation. As we are using AVRO schema for the data set and we don't have the use DT option there :(
Any alternatives for this?
Accepted Solution
Updated: 19 Apr 2024 4:02 EDT
ING Nederland
NL
We have an activity which is called at the end of the dataflow (data set -> convert shape -> activity).
In this activity we copy the primary page(which is the de-serialized page) to a temp page & then using a JSON DT with auto-map feature we again serialize the temp page to get the result in param.jsonData.
P.S. In the settings tab of the JSON DT exclude the OOTB properties such as pxObjClass etc.
Regards,
Saurabh.