Stream Service is stopped
The Stream service is stopping regularly , when timestamps are verified between servers then the following exceptions are found in Kafka server logs file . Attached the complete log file.
Seems to be trying to clean up the log file but unable to do it and then the stream service is getting stopped.
Once stream node is stopped then getting many exceptions in all nodes .
I found Pega article related to same problem but it Applies to Pega Platform™ version 8.5 through Pega Platform version 8.7 . we are on Pega 8.8.1
***Edited by Moderator Marije to add Support ticket tags***
Accepted Solution
Updated: 9 Nov 2023 8:31 EST
Pegasystems Inc.
GB
@BRAHMESH@ I have checked internally and was able to find another Pega 8.8 client who experienced the same errors.
If the stream node is running on windows server then you may be experiencing the issue described in Support Document Kafka fails to clean up compacted topics (Windows Server 2016)
Note that this is not Pega issue but kafka limitation on Windows.
The changes shown in the article should prevent this issue. Also note that for any reason if kafka-data folder is deleted from sesrver, you have to perform above changes.
If the above resolves your issue I will contact the author of the Support Document in order to ask that Pega 8.8 version be added to the 'applies to' list.
Skandiabanken
SE
@MarijeSchillern we had raised the ticket INC-A17719 for it ..it got solved after executing the script sfor clean up policy that mentioned in the article .
Ford Motor Company
US
@BRAHMESH@ Based on the log you have shared and the error messages, it appears the Stream Services are not stopped gracefully.
When Steam Services are not externalized and had a improper shutdown of the Steam Services, it wll start behaving differently, usually kafka try to create a topic but that topic already exist or running from other stream node , since it was an ungraceful shutdown, no control communication between the members of the Kafka cluster.
Perhaps, review following steps to troubleshoot the problem.
-- Stream Nodes Issue ---
Shutdown Stream Nodes [Make sure both Kafka and Pega JVM Stopped]
Truncate Table dbo.pr_data_stream_node_updates; Truncate Table dbo.pr_data_stream_nodes; Truncate Table dbo.pr_data_stream_sessions; Truncate Table dbo.pr_data_qp_run_partition;
Delete Kafka-Data
===
Stream Nodes Should be Stopped One at a Time. Stop Sequence is exact Opposite from Start Sequence Always One Stream Node should be up and running for Pega before any node type node can come up Web User Nodes always can be started last[Just best Practice]. Stopping doesnt matter. Search Nodes always one at a time .. Typically Primary Search Node followed by Secondary Search Node should start first. While stopping same Order can be followed.
Skandiabanken
SE
@SUMAN_GUMUDAVELLY I tried the steps that suggested but still getting the below exception in kafka server log.
we have only one stream node in the cluster but unable to figure out with this exception.
Cluster step
@SUMAN_GUMUDAVELLY I tried the steps that suggested but still getting the below exception in kafka server log.
we have only one stream node in the cluster but unable to figure out with this exception.
Cluster step
[2023-07-06 14:02:05,856] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PZSTANDARDPROCESSOR with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager) org.apache.kafka.common.errors.TopicExistsException: Topic 'PZSTANDARDPROCESSOR' already exists. [2023-07-06 14:02:05,919] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PZATTACHMENTMIGRATIONPROCESSOR with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager) org.apache.kafka.common.errors.TopicExistsException: Topic 'PZATTACHMENTMIGRATIONPROCESSOR' already exists. [2023-07-06 14:02:05,981] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PYPROCESSNOTIFICATION with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager) org.apache.kafka.common.errors.TopicExistsException: Topic 'PYPROCESSNOTIFICATION' already exists. [2023-07-06 14:02:06,041] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PYBIBATCHINDEXPROCESSOR with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager) org.apache.kafka.common.errors.TopicExistsException: Topic 'PYBIBATCHINDEXPROCESSOR' already exists. [2023-07-06 14:02:06,105] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic ASYNCPROCESSING with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager) org.apache.kafka.common.errors.TopicExistsException: Topic 'ASYNCPROCESSING' already exists. [2023-07-06 14:02:06,169] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic ASYNCPROCESSINGDELAYED with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager) org.apache.kafka.common.errors.TopicExistsException: Topic 'ASYNCPROCESSINGDELAYED' already exists. [2023-07-06 14:02:06,227] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PXINTERACTIONAGGREGATOR with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager) org.apache.kafka.common.errors.TopicExistsException: Topic 'PXINTERACTIONAGGREGATOR' already exists. [2023-07-06 14:02:06,280] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PYSENDEMAIL with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager) org.apache.kafka.common.errors.TopicExistsException: Topic 'PYSENDEMAIL' already exists.
Ford Motor Company
US
@BRAHMESH@ This Error is Indicative of corrupted data in the Folder: Kafka-Data, which needs to e cleared. Even after clearing the Kafka-Data and restart the JVM's, that requires additional debugging.