Pega CDH External Kafka Stream Service: newer version issue
I am configuring an externally managed stream server on kafka cluster 2.6.0, based on Confluent, for Pega CDH
Connection is working fine and the Stream Service is reported as Normal in the Decisioning->Infrastructure->Services->Stream Window.
Unfortunately, I get a lot of error over two issues:
1) For customer requirement, I need to set a prefix to the consumer groups and found no documentation about how to do it: the application tries to create the consumer groups df_run_metrics_subscription and vbd_insert_subscription and I can't find anywhere where they are defined
2) The customer reports that Pega tries to use the outdated API Cluster -> Create instead of Topic -> Create, which is mandatory. The new API is described in Kafka KIP-277
Can anyone help me on the topics above? I have been looking in the Collaboration center for two days: found a lot of useful stuff, but nothing about that.
Thanks
Filippo
Pegasystems Inc.
IN
Hi Phill,
For your query #1 does the below post help ?
https://collaborate.pega.com/question/getting-blank-messages-kafka-topic
Thanks
Updated: 10 Dec 2020 3:13 EST
Realvalue Consulting
IT
Dear Abhijit,
thank you for your kind response.
Unfortunately I already read that content, but did not change the scenario, also because I need to set up two different consumer group and not just a global one.
I was wondering where the name of those consumer group are defined in the CDH application, weather in some jar or in some database table.
Best Regards
Filippo
Pegasystems Inc.
IN
@Phil5873 Hi Phill,
Can you please elaborate further about your requirement?
I am trying to understand the following:-
- Which consumer and consumer groups are you referring here?
- As you are maintaining an external Kafka who are you referring to only Pega side consumers in this case ?
- Are you referring to custom real time data flows sourced by Kafka/Stream datasets as consumers are you trying to group? - It seems Pega does not use the concept of consumer group , as it keeps track of its position in the topic via the database rather than relying on Kafka. Also multiple consumer group creation is not supported.
https://collaborate.pega.com/question/pega-kafka-client-connector-consumer-group-support
Regards
Abhijit