Issue with Stream Node
We’re currently facing an issue where the Stream node is intermittently stopping in our environment. As part of our investigation, we referred to a https://support.pega.com/support-doc/kafka-fails-clean-compacted-topics-windows-server-2016? which suggests setting the cleanup.policy=delete on certain Kafka topics to prevent log retention issues.
Following that guidance, we attempted to update the cleanup policy using the following command:
D:\PEGA\CT5STREAM\Stream\kafka- Proprietary information hidden\bin\windows>kafka-topics.bat --zookeeper {IPAddressOfStreamNode}:2181 --topic __consumer_offsets --alter --config cleanup.policy=delete
We’re currently facing an issue where the Stream node is intermittently stopping in our environment. As part of our investigation, we referred to a https://support.pega.com/support-doc/kafka-fails-clean-compacted-topics-windows-server-2016? which suggests setting the cleanup.policy=delete on certain Kafka topics to prevent log retention issues.
Following that guidance, we attempted to update the cleanup policy using the following command:
D:\PEGA\CT5STREAM\Stream\kafka- Proprietary information hidden\bin\windows>kafka-topics.bat --zookeeper {IPAddressOfStreamNode}:2181 --topic __consumer_offsets --alter --config cleanup.policy=delete
ERROR StatusLogger Reconfiguration failed: No configuration found for '21588809' at 'null' in 'null' Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:225) at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:221) at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:221) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:250) at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:221) at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:95) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1538) at kafka.admin.TopicCommand$.main(TopicCommand.scala:57) at kafka.admin.TopicCommand.main(TopicCommand.scala)
However, the command fails to execute properly. We suspect it may be due to connectivity issues with ZooKeeper, Has anyone successfully updated topic configurations in Pega-managed Kafka (especially internal topics like __consumer_offsets)?
I tried this command while stream node is up and running.
ai4process ltd
GB
Internal topics like __consumer_offsets are embbedded and tightly controlled
Kafka versions post 2.8+ are moving away from ZooKeeper. If your Pega upgrade included Kafka 3.x, ZooKeeper-based commands may no longer work.
The support doc suggesting to add Clean-up policy is typically suggested for Custom or application-level topics and not internal ones.
It'd be better to understand why your stream nodes are intermittently stopping rather than updating the internal topics.
Stream nodes stopping may not be always due to kafka configuration, it may also be due to the lack of sufficient system resources (JVM Memory CPu, Heap size etc), or any Database schema connectivity -
Monitoring queries to the pr_data_stream_nodes, pr_data_stream_sessions, and pr_data_stream_node_updates tables can help identify database-related bottlenecks
Skandiabanken
SE
@RaviChandra I believe Pega is still using the embedded Kafka version kafka- Proprietary information hidden, and as far as I know, we do not have control to upgrade this embedded Kafka version ourselves.
We encountered the same Stream node issue during our previous upgrade in 2023. As part of the Pega SR investigation at that time, it was suggested that we set the cleanup.policy=delete for the topics mentioned in the related Pega article.https://support.pega.com/support-doc/kafka-fails-clean-compacted-topics-windows-server-2016?, . After our recent upgrade to 24.1.2, we have run into the same issue again. However, when I execute the same command to define the cleanup policy this time, it does not work.
I’m trying to understand whether there is anything wrong with the command syntax, or if something has changed that would prevent it from working after the latest upgrade