Error by switching from internal to external Kafka service
Hello,
I facing an issue by switching from an internal to external managed service.
When I configure the external kafka the pods for backend and frontend throws some exceptions see attachments.
Are there special configurations for the kafka services which pega needs?
The connection is configured in the helmchart, I've only use the following options for the connecion
stream:
enabled: true
bootstrapServer: "<domain_to_managed_kafka>"
Attached you find the logs of the two above mentioned pods.
Thank you
Accepted Solution
Updated: 28 Jul 2023 1:34 EDT
T-Systems International GmbH
DE
@SriHarsha Anika @MarijeSchillern
Problem is solved, the main problem was a missing port number at the end of the domain for the bootstrap server.
Changeing it to:
bootstrapServer: "kafka.mydomain.de:9092"
helps to get the Kafka connected.
Thank you
Julian
Updated: 13 Jun 2023 9:21 EDT
Pegasystems Inc.
GB
@JulianB70 `can you tell us what research you've carried out so far?
Which documentation have you followed so far?
Externalization of services in your deployment
Externalization for a Kafka service
Were any of the previous questions on the forum useful?
Configure External Kafka as a stream service
Externalizing Kafka possible on VMs?
Kafka configuration instance - properties file - error connecting to external Kafka
T-Systems International GmbH
DE
I followed this documentation: https://github.com/pegasystems/pega-helm-charts/blob/master/charts/pega/MigrationToExternalStream.md
The kafka has no User/password configured and Kafka SASL_SSL is not configured.
Domain to managed kafka service is reachable from the kubernetes cluster.
All used components running in a cloud.
JulianB70
Updated: 13 Jun 2023 9:24 EDT
Pegasystems Inc.
GB
@JulianB70 the error includes:
Caused by: org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in bootstrap.servers: kafka.extelastic1.bpmaas.de
I guess searching on those errors would be a good starting point.
Failing further responses on the forum I would suggest that you log a support incident on the MSP. Please let us know the INC reference if you chose to go down that route.
Updated: 28 Jul 2023 1:34 EDT
T-Systems International GmbH
DE
Thank you for your reply. I've a question, in which format did pega expect the URL for the bootstrap servers?
Here some examples:
JPMorgan Chase
IN
Setting should be set to false.
Input the external kafka configuration as part of prconfig.
stream:
enabled: false
Accepted Solution
Updated: 28 Jul 2023 1:34 EDT
T-Systems International GmbH
DE
@SriHarsha Anika @MarijeSchillern
Problem is solved, the main problem was a missing port number at the end of the domain for the bootstrap server.
Changeing it to:
bootstrapServer: "kafka.mydomain.de:9092"
helps to get the Kafka connected.
Thank you
Julian
Cognizant
ES
@JulianB70 Hello!, can you specify where you set the bootstrap servers? I set the HOST Url on the Kafka Rule, and Schema Registry URLs on the AVRO Schema... but I can't find where to set the bootstrap servers.
If I add them to the .properties file, I always get an error about "disallowed Kafka client configuration properties".
Thank you!