Question
Sunlife
IN
Last activity: 15 Feb 2024 6:55 EST
Pre-requisite for setting up external Kafka
We are planning to switch embedded kafka to external kafka, would like to understand any pre-requisite to handle below items.
1. How would kafka topics be created for the existing queue processors in the external kafka broker?
2. Does all the events in the embedded kafka topics moves automatically into external kafka topics?
3. Does number of partition in the topic created in external broker would still follow '20' partitions? I assume it can be overridden.
4. Would topic in external kafka, continue with same offset in the embedded kafka topics? If not , how do we reset the offset at the consumer?
Accepted Solution
Updated: 29 Feb 2024 8:14 EST
JPMorgan Chase
IN
Swedbank AB
SE
@SriHarsha Anika
Thanks for the explanation. We are also in the process to move towards externalizing the embedded services starting with Kafka and these answers are really helpful. However I had a follow-up question on this.
When we externalize kafka and connect to an external cluster ( confluent) , I believe the current file system ( kafka-data) will be created fresh in the external cluster. Will Pega continue to maintain the metadata, topic information in the stream DB tables? Or will these be handled in the external kafka clusters?
As in the current processing scenario, if the queuing fails for some reason the information is stored in the Pega DB and when the node is available it will pick up the queued messages from the DB. How will this scenario work when the topics are in teh external clusters?
@SARATHVISWA
https://docs.pega.com/bundle/platform-88/page/platform/deployment/externalization-of-services/externalize-kafka-in-your-deployment.html
1. Given sufficient administrative privilege to the Pega account which is interacting with kafka - Topics will be auto created
2. Existing events may not be migrated unless you leverage migration tools
3. partitions will as per the default settings provided during the kafka configuration
4. Offset will not be the same, it will be different as topics are created brand new with new infrastructure zookeeper starts freshly.