Discussion
Ikor
AU
Last activity: 18 May 2022 11:48 EDT
How to get offset and partition key after publishing a message on a kafka topic
Hi,
I am unable to get offset and partition number after publishing a message on a kafka topic. Can anyone help me how to get that?
Thanks in advance,
-
Like (0)
-
Share this page Facebook Twitter LinkedIn Email Copying... Copied!
Updated: 27 Apr 2022 16:38 EDT
Pegasystems Inc.
US
There is no built-in support for Apache Kafka in Pega 7. With that said, can you post your code showing what you are able to get?
For example :
for (kafka.consumer.KafkaStream<byte[], byte[]> stream : consumerMap.get(topic)) {
kafka.consumer.ConsumerIterator<byte[], byte[]> it = stream.iterator();
while (it.hasNext()){
String messageId = new String(it.next().message());
//process
}
}
consumer.shutdown();
Also, how are your properties set up?
props.put("partition.assignment.strategy", "range");
props.put("auto.offset.reset", "smallest");
Updated: 27 Apr 2022 16:38 EDT
Ikor
AU
Hi Paul,
Thanks for your reply.
I have followed below link and able to publish data in topic both way from data flow and also from activity. But I need offset and partition number after publishing the message, that I am not getting.
https://pdn.pega.com/kafka-data-sets-decision-management/kafka-data-sets-decision-management
Updated: 27 Apr 2022 16:38 EDT
Pegasystems Inc.
US
Read the doc and am testing on my 7.3 ref. system. From what I see so far, those two values are not part of the dataset. The Kafka Consumer API for reading a message also does not provide them - and that is what I suspect Pega is employing (as opposed to AdminClient). Will research further and reach out internally.
-
Deexith Bairy Sergiy Ladnych
Updated: 27 Apr 2022 16:38 EDT
Pegasystems Inc.
US
So, one of the SMEs suspects it might be available from Consumer API but we are not exposing it. I asked about options to discover and he stated its all internal code, so very difficult to access from any sort of local change. I have another SME resource who is part of the team working on the 7.4 modifications - so I'll reach out in that direction.
Updated: 27 Apr 2022 16:38 EDT
Pegasystems Inc.
US
This is not supported. Pega development is curious as to what use case you have that requires you to get offset and partition after publishing. Can you provide that?
Updated: 27 Apr 2022 16:38 EDT
Ikor
AU
While pushing a message on kafka topic, that message is stored in some partition and offset. I need that partition and offset number to check what actually posted from pega and also for our automation testing we need. We have another get rest service to check the posted message on topic using this offset and partition number.
-
Shrikant Pustode
Updated: 27 Apr 2022 16:38 EDT
Pegasystems Inc.
US
Thanks.
I will forward this use case description to the SME. I'll let you know if this generates an enhancement request and will provide the tracking number.
Updated: 27 Apr 2022 16:38 EDT
Ikor
AU
Thanks.
Updated: 27 Apr 2022 16:38 EDT
Pegasystems Inc.
US
An enhancement request has been generated for this use case. You can request updates by referring to FDBK-24065. If I hear further, I will update here as well.
Updated: 27 Apr 2022 16:38 EDT
Ikor
AU
Hi Paul,
Can you please give me some update on this?
Thanks & Regards,
Amar
Updated: 27 Apr 2022 16:38 EDT
Pegasystems Inc.
US
Hi Amar,
I'm not showing any update to this enhancement currently. Please reach out to your Account Executive for more information.
Thanks!
Updated: 27 Apr 2022 16:38 EDT
Rulesstack Solutions LTD
US
Hi Amar,
I am looking for similar use case to post messages to Kafka from Pega. Can you please help on any doc on how you were able to achieve this in 7.3.
My use case is to post in Kafka topic from Pega clipboard data in JSON format.
Updated: 27 Apr 2022 16:38 EDT
Ikor
AU
Please go though below link, hope it will help you
https://community.pega.com/knowledgebase/articles/kafka-data-sets-decision-management