Question
SOFTTECH A.Ş.
TR
Last activity: 13 Nov 2025 10:13 EST
External Kafka Connection Timeout issue when tomcat starts
Our client has deployed an external Kafka server to connect from on prem Pega 24.1.3 Dev environment and configured prconfig.xml file to connect to external kafka. When we start tomcat web server we get that error:
[ALIZE_STREAM_SERVICE] [ STANDARD] [ ] [ ] (.stream.StreamServiceValidator) ERROR - Failed to initialize StreamAPI: attempt 1 from 6. Next retry in 20 seconds
java.util.concurrent.TimeoutException: null
We can ping to External Kafka server from Pega Dev Node successfully, but we get time out error just in time we start Tomcat. Here are our Kafka related settings in PrConfig.xml :
<env name="services/stream/provider" value="ExternalKafka"/>
<env name="services/stream/broker/url" value=" Proprietary information hidden:9092"/>
<env name="services/stream/replication/factor" value="1"/>
<env name="services/stream/encryption/security/protocol" value="PLAINTEXT"/>
<env name="services/stream/encryption/sasl/mechanism" value="PLAIN"/>
<env name="services/stream/name/pattern" value="pega-dev-{stream.name}"/>
<env name="dnode/stream/node/disabled" value="true"/>
<env name="services/stream/kafka/request.timeout.ms" value="40000"/>
<env name="services/stream/kafka/metadata.max.age.ms" value="20000"/>
Our client has deployed an external Kafka server to connect from on prem Pega 24.1.3 Dev environment and configured prconfig.xml file to connect to external kafka. When we start tomcat web server we get that error:
[ALIZE_STREAM_SERVICE] [ STANDARD] [ ] [ ] (.stream.StreamServiceValidator) ERROR - Failed to initialize StreamAPI: attempt 1 from 6. Next retry in 20 seconds
java.util.concurrent.TimeoutException: null
We can ping to External Kafka server from Pega Dev Node successfully, but we get time out error just in time we start Tomcat. Here are our Kafka related settings in PrConfig.xml :
<env name="services/stream/provider" value="ExternalKafka"/>
<env name="services/stream/broker/url" value=" Proprietary information hidden:9092"/>
<env name="services/stream/replication/factor" value="1"/>
<env name="services/stream/encryption/security/protocol" value="PLAINTEXT"/>
<env name="services/stream/encryption/sasl/mechanism" value="PLAIN"/>
<env name="services/stream/name/pattern" value="pega-dev-{stream.name}"/>
<env name="dnode/stream/node/disabled" value="true"/>
<env name="services/stream/kafka/request.timeout.ms" value="40000"/>
<env name="services/stream/kafka/metadata.max.age.ms" value="20000"/>
If PrConfig.xml settings are wright, what may be wrong? Should we suspect from network related issues even though we can reach at external Kafka server from our Pega node from command line ?
Thanks
Pegasystems Inc.
US
@AHMETBURAKS Based on the description of your setup, my guess is there are some network firewall rules that block this traffic. Check with the netops team(s) to confirm the required ports are open for traffic in the network firewalls.
Updated: 13 Nov 2025 9:12 EST
SOFTTECH A.Ş.
TR
@PhilipShannon Thank you very much, i have just asked to our client's infrastructure team to check security restrictions, but they told that there is full access between Pega node and external kafka node. Actually, they showed me that Pega's requests are reaching at Kafka server and _consumer_offsets topic seems as created on Kafka data folder, but nothing else. I have attached full pega rules log, if you want to investigate.
Pegasystems Inc.
US
@AHMETBURAKS Also check the "-DNodeType=" setting for this system, it doesn't seem that it's configured as a stream node
2025-11-13 16:53:12,598 [ main] [ ] [ ] [ ] (nternal.NodeClassificationImpl) INFO - NodeTypes considered on current node = [BackgroundProcessing, WebUser] , for given -DNodeType = [ BackgroundProcessing,WebUser ]
within the logfile you can see that it's not trying to start any stream node
@AHMETBURAKS Also check the "-DNodeType=" setting for this system, it doesn't seem that it's configured as a stream node
2025-11-13 16:53:12,598 [ main] [ ] [ ] [ ] (nternal.NodeClassificationImpl) INFO - NodeTypes considered on current node = [BackgroundProcessing, WebUser] , for given -DNodeType = [ BackgroundProcessing,WebUser ]
within the logfile you can see that it's not trying to start any stream node
2025-11-13 16:54:41,943 [e-registry-heartbeat] [ STANDARD] [ ] [ ] (.tasks.TopologyChangedDetector) INFO - Services topology has changed. Current topology: This node ==> DataFlow.BackgroundProcessing edb548c27ce84230385384cbf6584938 Proprietary information hidden NORMAL 8.24.3 41eaa1b5-909b-4809-bd05-56ca13524d87 This node ==> DataFlow.WebUser edb548c27ce84230385384cbf6584938 Proprietary information hidden NORMAL 8.24.3 7c1a3fc8-0b0e-495d-9d00-a146ce2a583b This node ==> pega edb548c27ce84230385384cbf6584938 Proprietary information hidden running (was startingUp) 8.24.3 6cd8b9b9-c84a-4664-b432-bbd1323c0986
Try with this setting instead: -DNodeType=WebUser,BackgroundProcessing,Stream,Search
SOFTTECH A.Ş.
TR
@PhilipShannon Actually, i think node types you pointed refers to our Pega node, not external Kafka node; So, should we still add Stream,Search into it ?
2025-11-13 16:53:12,598 [ main] [ ] [ ] [ ] (nternal.NodeClassificationImpl) INFO - NodeTypes considered on current node = [BackgroundProcessing, WebUser] , for given -DNodeType = [ BackgroundProcessing,WebUser ]
Thank you,
Burak.
Pegasystems Inc.
US
@AHMETBURAKS That is going a bit outside of my area, it's possible that using external kafka doesn't require the stream node type.
But I do see this in your logfile:
java.util.concurrent.CompletionException: com.pega.fnx.stream.spi.StreamServiceException: Unable to connect to the stream provider. at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[?:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:842) [?:?] Caused by: com.pega.fnx.stream.spi.StreamServiceException: Unable to connect to the stream provider.
To fix this, I would add stream to the DNodeType and go from there, just to get the system up and running.