Question
Commonwealth Bank of Australia
AU
Last activity: 4 May 2020 3:54 EDT
How does multiple Keys for Avro Deserialization for Pega KAFKA ?
Hi,
The ability to do AVRO Serialization and Deserialization for t-Key currently is not available in PEGA KAFKA. The schema parser only applies to the t-Value. With that, the current consumers will fail since it expects to de-serialize the key AVRO format.
Since Pega also provides a way to specify multiple Partition key(s), how would t-Key AVRO SeDer works?
Please let me know how can we mention multiple keys if the Avro Key is expecting multiple keys.
Thanks,
KC
Pegasystems Inc.
US
To clarify, have you already implemented a custom deserializer for AVRO? That is required, see links: https://collaborate.pega.com/discussion/implementation-apache-kafka%C2%AE-apache-avro-serialization-pega
https://collaborate.pega.com/discussion/kafka-custom-serializerdeserializer-implementation
Commonwealth Bank of Australia
AU
Hi @KevinZheng_GCS ,
We have implemented this custom deserializer for AVRO. We have successfully able to consume the data from the topic and publish the data to topic.
We have the Key Schema along with the Value Schema for the topic. We are able to publish the data without Key from Pega. But incase, if the other consumer is expecting the Key Schema along with Value Schema (i.e., Key Schema is mandatory), then what should Pega consider ? We have observed, it is not taking the Partition Key as the Key Schema.The Key Schema could be compound / nested structure. As per the document provided, the Partition Key determines how the data from the Kafka topic will be spread it in between Pega partitions in the Pega Data Flow run to improve the processing time. Then how do we need to send the Key ? And what if the Key Schema is expecting multiple Keys . How do we handle this scenario?
Thanks,
KC