Question
Cognizant Technology
CA
Last activity: 9 Apr 2024 11:51 EDT
Pega support for Azure Event Hub instead of externalized Kafka for Azure Cloud choice
Hi Team
We couldnt find any official document for Pega support for Azure Event Hub instead of externalized Kafka for Azure Cloud choice . Can we please get an confirmation if Pega supports Azure event Hubs for Pega deployments on Azure AKS. or if these are not supported and the challenges with it.
Accepted Solution
Updated: 9 Apr 2024 11:51 EDT
Pegasystems Inc.
BR
There are many different flavors of Kafka nowadays and it’s very challenging for Pega to validate the external Stream service using all these flavors, so it was mostly validated with the main ones (Amazon MSK, Confluent Platform/Cloud, Instaclustr, Bitnami and Apache).
With that being said, I have helped a client to implement EventHub integration with their external stream service on Platform 8.8.2 version (also tested in 8.7.3) and it’s been working since then without issues. The only limitation is that Platform currently supports only PLAIN and SCRAM authentication for SASL mechanisms and MS EventHub is commonly used with OAuth 2.0.
Here’s some useful information that can help you in achieving a successful stream service integration with EventHub using PLAIN authentication (the following information is considering default settings for a new EH namespace):
External Stream integration with MS EventHub
Every EventHub cluster created using default settings utilize the following ports:
There are many different flavors of Kafka nowadays and it’s very challenging for Pega to validate the external Stream service using all these flavors, so it was mostly validated with the main ones (Amazon MSK, Confluent Platform/Cloud, Instaclustr, Bitnami and Apache).
With that being said, I have helped a client to implement EventHub integration with their external stream service on Platform 8.8.2 version (also tested in 8.7.3) and it’s been working since then without issues. The only limitation is that Platform currently supports only PLAIN and SCRAM authentication for SASL mechanisms and MS EventHub is commonly used with OAuth 2.0.
Here’s some useful information that can help you in achieving a successful stream service integration with EventHub using PLAIN authentication (the following information is considering default settings for a new EH namespace):
External Stream integration with MS EventHub
Every EventHub cluster created using default settings utilize the following ports:
- 9092 - for inter-broker communication (insecure/not exposed) considering there are 3 AZs providing data redundancy.
- 9093 - for client connections. It is SSL-secured (doesn’t require client-side truststore by default) and exposed over internet by default.
- 443 - for API calls, SchemaRegistry over HTTPS and Service bus operations.
To establish a connection to the EventHub endpoint, a Kafka principal needs to be created which Microsoft named as SharedAccessKey. This SharedAccessKey needs to be created with MANAGE permission (Microsoft call resource permissions as claims) which includes LISTEN and SEND operations (similar level of access when ALL operations are given in Kafka ACL authorizer mechanism).
To create an SharedAccessKey, it can be done via Azure portal > EventHub > All services > your-eventhub-namespace > Settings > Shared access policies.
Once the SharedAccessKey is created, we need to copy the Connection string–primary key which will be used as the password for authentication when constructing the Kafka producer/consumer later. The Connection string–primary key will look like below example:
Endpoint=sb://bruno11ce.servicebus.windows.net/;SharedAccessKeyName=Principal;SharedAccessKey=IYfC+5I0+SpczMiVyXaC2NdacP8S2+OcK+AEhGBOdUM=
EventHub’s authentication does accept SASL PLAIN security mechanism with SSL encryption, so when configuring the client properties to convert them into PRCONFIGs, the security protocol and mechanism should be:
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
Since it’s using PLAIN, a username and password needs to be provided for the Plain LoginModule. For EventHub, the Connection string that was mentioned earlier contain the necessary strings for both username and password (SharedAccessKeyName & SharedAccessKey), but EventHub is expecting the credentials to be passed using a $ConnectionString mapping for the username string which is parsed on server side during the authentication challenges, so we need to configure the SASL JAAS LOGIN client property like below:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username="$ConnectionString" password="Endpoint=sb://bruno11ce.servicebus.windows.net/;SharedAccessKeyName=Principal;SharedAccessKey=IYfC+5I0+SpczMiVyXaC2NdacP8S2+OcK+AEhGBOdUM=";
Since EventHub uses 3 availability zones by default, the replication factor must be set to 3.
There are no prefixed permissions for topic names (Microsoft call topics as “event hubs”) – so any stream name pattern can be used with MANAGE permission/claim and Platform’s commands to create topics should succeed.
Here are the external stream PRCONFIGs that were used for our internal test, as an example (use single quotes for the username and password):
<env name="services/stream/provider" value="ExternalKafka"/>
<env name="services/stream/broker/url" value="bruno11ce.servicebus.windows.net:9093"/>
<env name="services/stream/external/replication/factor" value="3"/>
<env name="services/stream/encryption/security/protocol" value="SASL_SSL"/>
<env name="services/stream/name/pattern" value="pega873-{stream.name}"/>
<env name="services/stream/encryption/sasl/mechanism" value="PLAIN"/>
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.kafka.common.security.plain.PlainLoginModule required
username='$ConnectionString'
password='Endpoint=sb://bruno11ce.servicebus.windows.net/;SharedAccessKeyName=Principal;SharedAccessKey=IYfC+5I0+SpczMiVyXaC2NdacP8S2+OcK+AEhGBOdUM=';" />
Considering the above information/instructions, once Platform is started all topics/event hubs should be automatically created. Even though the EventHub portal will display topics in lower case, they are actually created using Platform’s logic (QP topics in upper case, stream data sets in camel case and internal topics in lower case).
As Platform needs to create more than 10 topics/event hubs per Pega instance, a Premium plan is required, otherwise there should be a limitation of only 10 topics per namespace and “PolicyViolationExceptions” are expected to happen.
In case you have the intention to point multiple Pega instances to the same EventHub namespace, make sure their corresponding stream name patterns (PRCONFIG “services/stream/name/pattern”) have unique prefixes/suffixes to avoid conflicting resources.