Kafka Cluster Reuse across Pega Deployments
Hello,
As part of our externalization effort, I would like to understand/retrieve information around Kafka Cluster reuse across multiple Pega workloads/Deployments.
Looking for some reference links/best practices around reusing kafka clusters across multiple deployments towards externalization the stream/kafka service and for cost savings.
I would like to review/understand best practices reference links and/or risks (if any) with respect to reusing kafka cluster across multiple pega deployments.
Pegasystems Inc.
GB
@SethuS88 Containerized deployments support sharing a single Kafka service between multiple Pega Platform deployments, where you have configured the Kafka service for multi-tenancy support.
To do so, configure a unique topic prefix in the Pega Helm chart for each deployment, which provides a unique configuration that avoids topic name conflicts when sharing a cluster. For more information about the required minimum settings for configuring your organization's Kafka service, see Pega requirements and guidelines for your Kafka service cluster.
For supported Kafka versions running in your Kafka service, see the Platform Support Guide. It is important to follow best practices for security, authentication, and authorization when configuring your Kafka cluster for multiple deployments.
This answer came from a Pega gen-AI assistant using the following references
Externalization for a Kafka service
Third-party externalized services Deployment Changes FAQs
Manheim
US
@MarijeSchillern How do we specify topic prefix for VM Based Deployments? In other words, if I have multiple pega deployments/applications (All of these are VM Based Deployments) trying to connect and reuse the same kafka cluster(shared cluster that is externalized), how can do this ?
Are these any specific DSS/Settings that can be applied to denote the topic prefix per Pega Deployment?
Also, what if we have to reuse the same cluster for multiple environments (dev/qa/staging/prod) that belongs to the same pega application/deployment?
Ford Motor Company
US
@SethuS88 There are several things that comes to play for Externalization. Most important one what type of Authentication you will be using for Externalized Kafka Services, if you are referencing Platform Kafka, you can say External Stream Services rather than External Kafka.
External Kafka refers the Connection to/from Kafka Server [Data instance of Kafka Rule].
Also which version of Pega you or on or you will be on.
Pega Infinity'23 does support OAuth 2.0 through Kafka data instance rules as it's using Kafka-clients 3.4.0 version. For externalized Stream context, Product Engineering team will be working on enhancement request (FDBK-104840) to include support for OAuth 2.0 possibly in Pega Infinity 23.1.1 version
Pega using Kafka-clients version 2.1.1 in all currently available versions of Pega Platform [ till 8.8.x]. This version does not have full native support for OAuth 2.0 (hence Confluent Cloud documentation suggest using Kafka client 3.2.1 or later versions), but there are some backwards compatibilities which might work with OAuth 2.0 with some specific client properties. Since the Kafka configuration instance rule (Data-Admin-Kafka instance) doesn’t accept SASL OAUTHBEARER specific client properties.
In case if you are looking for UserID and Password authentication, you can use below specified DSS.
@SethuS88 There are several things that comes to play for Externalization. Most important one what type of Authentication you will be using for Externalized Kafka Services, if you are referencing Platform Kafka, you can say External Stream Services rather than External Kafka.
External Kafka refers the Connection to/from Kafka Server [Data instance of Kafka Rule].
Also which version of Pega you or on or you will be on.
Pega Infinity'23 does support OAuth 2.0 through Kafka data instance rules as it's using Kafka-clients 3.4.0 version. For externalized Stream context, Product Engineering team will be working on enhancement request (FDBK-104840) to include support for OAuth 2.0 possibly in Pega Infinity 23.1.1 version
Pega using Kafka-clients version 2.1.1 in all currently available versions of Pega Platform [ till 8.8.x]. This version does not have full native support for OAuth 2.0 (hence Confluent Cloud documentation suggest using Kafka client 3.2.1 or later versions), but there are some backwards compatibilities which might work with OAuth 2.0 with some specific client properties. Since the Kafka configuration instance rule (Data-Admin-Kafka instance) doesn’t accept SASL OAUTHBEARER specific client properties.
In case if you are looking for UserID and Password authentication, you can use below specified DSS.
Pega-Engine • prconfig/services/stream/encryption/sasl/jaas/config/default RS:Pega-ProcessCommander Pega-Engine • prconfig/services/stream/name/pattern/default RS:Pega-ProcessCommander
Sample Value: topicname-env-{stream.name} Dev: topicname-dev-{stream.name} QA: topicname-qa-{stream.name} Prod: topicname-prod-{stream.name}
Pega-Engine • prconfig/services/stream/broker/url/default RS:Pega-ProcessCommander
Pega-Engine • prconfig/services/stream/provider/url/default RS:Pega-ProcessCommander
Pega-Engine • prconfig/services/stream/encryption/sasl/mechanism/default RS:Pega-ProcessCommander
Pega-Engine • prconfig/services/stream/encryption/security/protocol/default
Pega-Engine • prconfig/services/stream/external/replication/factor/default Default Value: 3
Pega-Engine • prconfig/services/stream/provider/default Default Value: ExternalKafka