@KrishnaK6257

There are many different flavors of Kafka nowadays and it’s very challenging for Pega to validate the external Stream service using all these flavors, so it was mostly validated with the main ones (Amazon MSK, Confluent Platform/Cloud, Instaclustr, Bitnami and Apache).

With that being said, I have helped a client to implement EventHub integration with their external stream service on Platform 8.8.2 version (also tested in 8.7.3) and it’s been working since then without issues. The only limitation is that Platform currently supports only PLAIN and SCRAM authentication for SASL mechanisms and MS EventHub is commonly used with OAuth 2.0.

Here’s some useful information that can help you in achieving a successful stream service integration with EventHub using PLAIN authentication (the following information is considering default settings for a new EH namespace):
 

External Stream integration with MS EventHub

Every EventHub cluster created using default settings utilize the following ports:

• 9092 - for inter-broker communication (insecure/not exposed) considering there are 3 AZs providing data redundancy.
• 9093 - for client connections. It is SSL-secured (doesn’t require client-side truststore by default) and exposed over internet by default.
• 443 - for API calls, SchemaRegistry over HTTPS and Service bus operations.

To establish a connection to the EventHub endpoint, a Kafka principal needs to be created which Microsoft named as SharedAccessKey. This SharedAccessKey needs to be created with MANAGE permission (Microsoft call resource permissions as claims) which includes LISTEN and SEND operations (similar level of access when ALL operations are given in Kafka ACL authorizer mechanism).

To create an SharedAccessKey, it can be done via Azure portal > EventHub > All services > your-eventhub-namespace > Settings > Shared access policies.

Once the SharedAccessKey is created, we need to copy the Connection string–primary key which will be used as the password for authentication when constructing the Kafka producer/consumer later. The Connection string–primary key will look like below example:

Endpoint=sb://bruno11ce.servicebus.windows.net/;SharedAccessKeyName=Principal;SharedAccessKey=IYfC+5I0+SpczMiVyXaC2NdacP8S2+OcK+AEhGBOdUM=

security.protocol=SASL_SSL
sasl.mechanism=PLAIN

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username="$ConnectionString" password="Endpoint=sb://bruno11ce.servicebus.windows.net/;SharedAccessKeyName=Principal;SharedAccessKey=IYfC+5I0+SpczMiVyXaC2NdacP8S2+OcK+AEhGBOdUM=";

There are no prefixed permissions for topic names (Microsoft call topics as “event hubs”) – so any stream name pattern can be used with MANAGE permission/claim and Platform’s commands to create topics should succeed.

Here are the external stream PRCONFIGs that were used for our internal test, as an example (use single quotes for the username and password):

<env name="services/stream/provider" value="ExternalKafka"/>
<env name="services/stream/broker/url" value="bruno11ce.servicebus.windows.net:9093"/>
<env name="services/stream/external/replication/factor" value="3"/>
<env name="services/stream/encryption/security/protocol" value="SASL_SSL"/>
<env name="services/stream/name/pattern" value="pega873-{stream.name}"/>
<env name="services/stream/encryption/sasl/mechanism" value="PLAIN"/>
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.kafka.common.security.plain.PlainLoginModule required
            username='$ConnectionString'
password='Endpoint=sb://bruno11ce.servicebus.windows.net/;SharedAccessKeyName=Principal;SharedAccessKey=IYfC+5I0+SpczMiVyXaC2NdacP8S2+OcK+AEhGBOdUM=';" />

As Platform needs to create more than 10 topics/event hubs per Pega instance, a Premium plan is required, otherwise there should be a limitation of only 10 topics per namespace and “PolicyViolationExceptions” are expected to happen.

In case you have the intention to point multiple Pega instances to the same EventHub namespace, make sure their corresponding stream name patterns (PRCONFIG “services/stream/name/pattern”) have unique prefixes/suffixes to avoid conflicting resources.