Question
HCL Technologies
NL
Last activity: 15 May 2025 3:57 EDT
How to send a kafka event when there is a . in the field name
I have to send an event to a Kafka topic using Avro schema and one of the field in the schema has a . in the name. Now Pega does not let me create properties with . in them. Can someone let me know how I can create a page and use Data-Set Execute to publish this kafka message to the topic? Tried using property qualifier pzExternalName but it is getting ignored. Is that useful only for REST connections? I tried converting the clipboard page to json to see if the property qualifier adds the . in the field name, but it did not.
***Edited by Moderator Marije to add Support Case INC-B27539; Awaiting GCS FDBK item ***
***Edited by Moderator Marije to add Support Case INC-B27539; added linked FDBK-116092 item ***
Accepted Solution
Updated: 31 Jul 2024 4:16 EDT
Pegasystems Inc.
GB
@Vidya Menon the product team has logged enhancement request FDBK-116092 for this issue.
---------------------------------------
- Description
-
The client has a requirement to send an event to a Kafka topic using an Avro schema. They created a data set and are using the DataSet-Execute method. In the Avro schema, one field contains a dot (.) in its name, but Pega does not allow creating a property with a dot in it. They tried configuring it using the pzExternalName property qualifier, but it didn't work.
It appears that the Kafka data set parsing mechanism does not utilize the pzExternalName qualifier. This qualifier only applies to legacy JSON parsing/streaming for REST connectors and services. To achieve the required functionality, we suggested that the client leverage a custom serializer/deserializer class.
The client has requested a product enhancement to consider the property qualifier for this use case.
--------------------------------------
Any users interested in this future feature should like this post and contact their Account Executive if they wish to discuss the FDBK-116092 further.
Updated: 22 Jul 2024 6:09 EDT
Pegasystems Inc.
GB
@Vidya Menon In Pega, property names cannot contain a period (.) as it is a reserved character used to denote page context. The pzExternalName property qualifier is used for mapping JSON data to properties in the data model that is generated from REST integration. It is not applicable for Kafka Avro schemas. For Kafka Avro schemas, you need to ensure that the property names in Pega match the field names in the Avro schema. If the Avro schema has a field with a period, you might need to modify the Avro schema to replace the period with an allowed character, such as an underscore (_), and use the same modified name for the corresponding property in Pega.
⚠ This is a GenAI-powered tool. All generated answers require validation against the provided references.
Conversion of ClipboardPage to GenericRecord has failed trying to publish message kafka avro schema
Implementation of the Apache Kafka® with the Apache AVRO serialization in Pega
Pega limitation of not allowing property with same name but different case - can
@Vidya Menon In Pega, property names cannot contain a period (.) as it is a reserved character used to denote page context. The pzExternalName property qualifier is used for mapping JSON data to properties in the data model that is generated from REST integration. It is not applicable for Kafka Avro schemas. For Kafka Avro schemas, you need to ensure that the property names in Pega match the field names in the Avro schema. If the Avro schema has a field with a period, you might need to modify the Avro schema to replace the period with an allowed character, such as an underscore (_), and use the same modified name for the corresponding property in Pega.
⚠ This is a GenAI-powered tool. All generated answers require validation against the provided references.
Conversion of ClipboardPage to GenericRecord has failed trying to publish message kafka avro schema
Implementation of the Apache Kafka® with the Apache AVRO serialization in Pega
Pega limitation of not allowing property with same name but different case - can
Property Qualifier pzExternalName missing in PEGA Platform 8.8.0
Didn't find pzExternalName Qualifier in 7.1.7 version
*** Edit **
I see that in INC-B27539 on the 18th July you were told the following:
From GCS:
".. we checked the use case with an SME, and it appears that the Kafka data set parsing mechanism does not use the pzExternalName qualifier. As you suspected, this property qualifier only applies to legacy JSON parsing/streaming for REST connectors and services.
To achieve the required functionality, you will have to rely on custom serializer/deserializer class
Refer to below link for guidance on implementing your own serializer/deserializer classes.
https://support.pega.com/discussion/kafka-custom-serializerdeserializer-implementation
https://github.com/pegasystems/dataset-integrations/tree/master/kafka-dataset-samples
"
We are waiting for GCS to pass on the FDBK item once they have logged it with Engineering
********************
HCL Technologies
NL
@MarijeSchillern Thanks for the reply. Unfortunately changing the avro schema of the source system is not a viable option for us. I have raised an SR requesting for an enhancement to support this feature.
Accepted Solution
Updated: 31 Jul 2024 4:16 EDT
Pegasystems Inc.
GB
@Vidya Menon the product team has logged enhancement request FDBK-116092 for this issue.
---------------------------------------
- Description
-
The client has a requirement to send an event to a Kafka topic using an Avro schema. They created a data set and are using the DataSet-Execute method. In the Avro schema, one field contains a dot (.) in its name, but Pega does not allow creating a property with a dot in it. They tried configuring it using the pzExternalName property qualifier, but it didn't work.
It appears that the Kafka data set parsing mechanism does not utilize the pzExternalName qualifier. This qualifier only applies to legacy JSON parsing/streaming for REST connectors and services. To achieve the required functionality, we suggested that the client leverage a custom serializer/deserializer class.
The client has requested a product enhancement to consider the property qualifier for this use case.
--------------------------------------
Any users interested in this future feature should like this post and contact their Account Executive if they wish to discuss the FDBK-116092 further.
Macquarie
AU
@MarijeSchillern We are not using the pzExternalName qualifier as the name of the element matches with the avro schema.
But still we are not able to push a kafka message in avro format to a kafka topic, able to send the JSON message but not the avro message.
Is there something which I am missing to be done in Pega to serialize the json into avro ?
Getting following exception :
(@baseclass)Page failed to save with message: org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
Pegasystems Inc.
GB
@AdityaK2715 please post a new question.