Question
Schwab
US
Last activity: 7 Sep 2023 21:52 EDT
Kafka configuration instance - properties file - error connecting to external Kafka
Hello all,
I am getting an error when trying to create a Kafka instance to connect to an external Kafka broker.
Pega 8.7.3
Need to use username/password.
I create new Kafka instance, enter Host/Port, upload Configuration properties file, click Test connectivity.
After 60 seconds of wait cursor - I get "Connection failed Error connection to Kafka.." (out of a java step in Data-Admin-Kafka.pzCheckConnection).
I've tried different versions of the properties file - here is one:
sasl.mechanism=PLAIN security.protocol=SASL_PLAINTEXT sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="ggggg" \ password="ggggg";
Also tried: sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="gggg" password="gggg"; security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN
No matter what I pass in username/password - it spins one minute and shows that error (it even says Timeout expired while fetching topic metadata).
Am I using incorrect format or properties in that configuration file?
Thank you in advance,
Bogdan
***Edited by Moderator Marije to add Support Case Details; ***
Accepted Solution
Pegasystems Inc.
GB
@BogdanD16582931 I can see that you logged support ticket INC-270302 for this.
The investigation showed you were introducing a Data-Admin-Kafka rule which used SASL authentication, but while testing the connectivity a generic message was happening suggesting a timeout scenario or error while collecting the topics metadata, which indicated that either authentication was failing or some parameters that the external brokers were expecting was not being passed as part of the client properties on Pega side.
Pega GCS provided a standalone Kafka application that has maximum level of debugging enabled on all Apache Kafka classes. Using this approach we were able to identify that the connection was happening but it was failing at the very first stage of SASL authentication challenges (SEND_APIVERSIONS_REQUEST) and then it was getting immediately disconnected.
The issue was resolved by introducing additional Kafka client properties which were expected on external Kafka side (SSL encryption with truststore certificate) and modifying the security protocol (from SASL_PLAINTEXT to SASL_SSL).
You responded that the connect issue was resolved - solution was to use SASL_SSL security protocol and specify a truststore with valid certificate.
I will mark this issue with the above Accepted Solution.
Corner SA
IT
@MarijeSchillernCan you share the property files that you used at the end?
Ford Motor Company
US
You can refer to below DSS or prconfig defined at https://docs-previous.pega.com/decision-management/87/configuring-external-kafka-stream-service
Pega-Engine • prconfig/services/stream/encryption/sasl/jaas/config/default RS:Pega-ProcessCommander Pega-Engine • prconfig/services/stream/name/pattern/default RS:Pega-ProcessCommander
Sample Value: topicname-env-{stream.name} Dev: topicname-dev-{stream.name} QA: topicname-qa-{stream.name} Prod: topicname-prod-{stream.name}
Pega-Engine • prconfig/services/stream/broker/url/default RS:Pega-ProcessCommander
Pega-Engine • prconfig/services/stream/provider/url/default RS:Pega-ProcessCommander
Pega-Engine • prconfig/services/stream/encryption/sasl/mechanism/default RS:Pega-ProcessCommander
Pega-Engine • prconfig/services/stream/encryption/security/protocol/default
Pega-Engine • prconfig/services/stream/external/replication/factor/default Default Value: 3
Pega-Engine • prconfig/services/stream/provider/default Default Value: ExternalKafka
Corner SA
IT
@SUMAN_GUMUDAVELLY we have already configured the external Kafka, here the problem is with a new Kafka Instance: Records > SysAdmin > Kafka, the related documentation can be seen here: Creating a Kafka configuration instance (pega.com). I am interested in having the client properties configuration file used by @BogdanD16582931 to solve the issue above.
Ford Motor Company
US
So you do not want External Stream and Looking for Kafka data instance connection.
For confluent, please use below content for your property file.
sasl.mechanism=PLAIN security.protocol=SASL_SSL sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="UserName" \ password="password";
If you are connecting Hadoop based kafka, then use following properties,
security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name=kafka sasl.jaas.config=com.ibm.security.auth.module.Krb5LoginModule required \ credsType=both \ useDefaultCcache=false \ useKeytab="location of .keytab file" \ principal="haddop principle";