Kafka com.pega.dsm.kafka.impl.RetryableOperationException SSL handshake error
Kafka: Receiving com.pega.dsm.kafka.impl.RetryableOperationException top class exceptions in our non-prod kafka cluster on multiple namespaces. What causes the Kafka SSL handshake error: current status ?
HCA Healthcare
US
UHG
US
@Sairohith Thanks for the information, I didn't supply enough details since I wasn't sure where this would go. Sharing more details.
We have 7 different pega environments using pega versions 23.1.2 24.2.1 and 24.2.2 on kubernetes that were working fine for months, sharing the same kafka cluster until july 29, and 30, where they all started sharing the above error at different times over those two days, roughly every two minutes. I'm not aware of any changes on those days. On two of these systems the error went away after about 5 days. I'm not sure why it started or went away. We tried new pega QP certs on one that eventually went away, but not on the other that went away. New certs on another still bad system, didn't cure it. Pega seems to still be working, so I'm thinking it could be relating to some custom pega kafka additions outside of QP's. Its almost like something has corrupted our external kafka cluster, since this happened to all the pega namespaces using just this in this one external kafka cluster over 48 hours.
I'm not sure how to find or clear this.
HCA Healthcare
US
given every Pega namespace (across 23.1.2/24.2.1/24.2.2 on k8s) hit SSL handshakes against the same external Kafka over July 29–30, then some “self-healed” ~5 days later, this smells like a cluster-side certificate/PKI path or network control issue rather than app code. Here’s a focused way to fix it. From a pod in a failing namespace, run openssl s_client -connect <broker-host>:<ssl-port> -servername <broker-host> -showcerts to confirm the broker serves a full chain and the SAN matches the DNS you use; if you see “unable to get local issuer” or the cert chain stops at an intermediate, rebuild the broker keystore to include full chain (leaf + all intermediates) and restart brokers. If openssl is clean, enable TLS debug once on a failing Pega pod (JAVA_TOOL_OPTIONS: -Djavax.net.debug=ssl,handshake) and watch for “PKIX path building failed,” “certificate_unknown,” or “no protocols in common”; map: PKIX = missing CA/intermediate in client truststore, certificate_unknown = SNI/hostname mismatch, no protocols = TLS/cipher mismatch. Compare a healthy vs failing namespace: confirm identical truststores (JKS vs PKCS12, same password, same CA set) mounted and referenced in the Kafka DSS (security.protocol, ssl.truststore.location/type/password, ssl.endpoint.identification.algorithm). Because two envs recovered after ~5 days, check OCSP/CRL: if your corporate JVMs have revocation checks on, an OCSP responder/CDN flap can create periodic handshakes every reconnect; verify outbound to AIA/OCSP URLs is allowed, or temporarily disable revocation (test only: -Dcom.sun.net.ssl.checkRevocation=false) to confirm, then fix egress. Check broker advertised.listeners: any recent DNS/ingress change can break SNI if clients connect via a name not in the cert; standardize clients to the cert’s DNS or reissue certs with all listener hostnames. Verify all broker nodes’ keystores were rotated consistently—mixed leaf certs or mixed TLS versions cause intermittent failures as clients rebalance; align ssl.keystore, ssl.truststore, and listeners on every broker and roll the cluster. Rule out infra changes on Jul 29–30: WAF/IDS/IPS or load balancer TLS policy updates often block specific ciphers; ensure brokers and JVMs support a common TLSv1.2+ cipher set. Finally, restart Pega Stream/QueueProcessor pods after truststore changes to flush stale connections, and watch cadence: ~120s failures usually line up with Kafka client reconnect/backoff; once the underlying TLS issue is corrected (chain/SAN/revocation/cipher), the RetryableOperationException stops without app code changes.
@IvanP276
@IvanP276
to stop Kafka SSL handshakes causing com.pega.dsm.kafka.impl.RetryableOperationException in non-prod. First, confirm you’re actually hitting an SSL listener (security.protocol = SSL or SASL_SSL) and not pointing an SSL client at a PLAINTEXT port or the wrong advertised.listeners. Next, verify the broker cert: not expired, correct SAN for the hostname you call, and full chain present (include intermediates); most handshakes fail on SAN mismatch or missing intermediates. Ensure client truststore contains the issuing CA(s) for the broker cert; if you use mutual TLS, also load the client keystore with the private key and cert chain. Align protocols/ciphers: Java on Pega nodes and brokers must support a common TLS version (TLSv1.2+), and corporate policies sometimes disable older ciphers causing silent fails. Check endpoint identification: if your cert CN/SAN doesn’t match DNS, either fix the cert or temporarily set ssl.endpoint.identification.algorithm to empty (better to fix cert). Validate passwords/paths for ssl.truststore/ssl.keystore and types (JKS vs PKCS12); wrong type or password throws handshake errors. Check clock skew on nodes; big skew can break cert validation. If using SASL_SSL, confirm mechanism and JAAS config are correct before the TLS step. On the cluster, verify listener.name.*.ssl.keystore/truststore settings and that brokers reload after cert changes. From a stream node, run an openssl s_client against the broker host:port to see the exact TLS error and cert chain. In Pega, restart the Stream/Kafka client tier after updating DSS/keystores and purge stale connection pools. Finally, scan broker and client logs for “PKIX path building failed,” “certificate_unknown,” or “no suitable protocol” to pinpoint whether it’s trust, identity, or cipher mismatch the fix will map directly to that message.