Question
Areteans
NZ
Last activity: 17 Feb 2022 23:44 EST
Issue stating PE 851 due to kafka server issues
I quite often encounter issues when starting my personal edition 8.5.1 due to kafka related issues as per tomcat console. Attaching the latest log from kafka folder for any suggestions on this behavior and its resolution.
During such occurrences as the instance doesn't come up I connect to the database and try to clear the tables-pr_data_stream_nodes,pr_data_stream_sessions and the instance comes up. However any suggestions to avoid this behaviour would be helpful as this has been a recurring issue.
Any comments or suggestions are appreciated.
Thanks,
Satish
-
Likes (1)
Pawel Adamczuk -
Share this page Facebook Twitter LinkedIn Email Copying... Copied!
Accepted Solution
Updated: 17 Feb 2022 23:44 EST
Pegasystems Inc.
GB
The below workaround was something I found which may be used to stabilize the affected environment, but cannot be considered as a permanent fix (and I don't have access to the jar files listed).
- When the platform starts it unpacks Kafka to the
kafka-1.1.0.X
folder, and then spins the Kafka process using the libraries from that folder. - For the following procedure the
kafka-1.1.0.X
folder should already exist on the disk.- Move out from the kafka-1.1.0.X/lib folder two jars:
kafka_2.11- Proprietary information hidden.jar
andkafka-clients- Proprietary information hidden.jar
- Copy into the kafka-1.1.0.X/lib folder two patched jars
kafka_2.11- Proprietary information hidden.jar
andkafka-clients- Proprietary information hidden.jar
from thekafka- Proprietary information hidden
distribution.
- Move out from the kafka-1.1.0.X/lib folder two jars:
- Restart Stream service or the node.
Updated: 14 Feb 2022 9:18 EST
Pegasystems Inc.
GB
@SatishKumarL5551 could you confirm your version of Kafka?
Your error extract
---------------------------------------
ERROR Failed to clean up log for __consumer_offsets-0 in dir C:\PRPC851\PRPCPersonalEdition\tomcat\kafka-data due to IOException (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: C:\PRPC851\PRPCPersonalEdition\tomcat\kafka-data\__consumer_offsets-0\00000000000000000000.log.cleaned: The process cannot access the file because it is being used by another process.
at java.nio.file.Files.deleteIfExists(Files.java:1165) ~[?:1.8.0_121] at kafka.log.Cleaner.deleteCleanedFileIfExists$1(LogCleaner.scala:488) ~[kafka_2.11- Proprietary information hidden.jar:?]
----------------------------
Kafka has known issues on windows environment.
Pega did addressed this issue by update the kafka version.
The kafka version kafka_2.11- Proprietary information hidden, which is default in Pega 8.5.4 have taken care of the issues.
There is a local change. However my recommendation will be to wait and apply if the issue is frequent.
Until 8.6 Pega has uses two compacted topics: __consumer_offsets and df_run_metrics. Covert these two topics to the not compacted topics:
@SatishKumarL5551 could you confirm your version of Kafka?
Your error extract
---------------------------------------
ERROR Failed to clean up log for __consumer_offsets-0 in dir C:\PRPC851\PRPCPersonalEdition\tomcat\kafka-data due to IOException (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: C:\PRPC851\PRPCPersonalEdition\tomcat\kafka-data\__consumer_offsets-0\00000000000000000000.log.cleaned: The process cannot access the file because it is being used by another process.
at java.nio.file.Files.deleteIfExists(Files.java:1165) ~[?:1.8.0_121] at kafka.log.Cleaner.deleteCleanedFileIfExists$1(LogCleaner.scala:488) ~[kafka_2.11- Proprietary information hidden.jar:?]
----------------------------
Kafka has known issues on windows environment.
Pega did addressed this issue by update the kafka version.
The kafka version kafka_2.11- Proprietary information hidden, which is default in Pega 8.5.4 have taken care of the issues.
There is a local change. However my recommendation will be to wait and apply if the issue is frequent.
Until 8.6 Pega has uses two compacted topics: __consumer_offsets and df_run_metrics. Covert these two topics to the not compacted topics:
<Check your kafka version>
$./<kafka- Proprietary information hidden>/bin/kafka-topics.sh --zookeeper localhost:2181 --topic __consumer_offsets --alter --config cleanup.policy=delete $./<kafka- Proprietary information hidden>/bin/kafka-topics.sh --zookeeper localhost:2181 --topic df_run_metrics --alter --config cleanup.policy=delete
Areteans
NZ
@MarijeSchillern Thanks for the prompt response. Based on kafka folder name(kafka- Proprietary information hidden) in the PE installation, its version as well is Proprietary information hidden
Please advise accordingly.
Pegasystems Inc.
GB
@SatishKumarL5551 the issue caused by the Stream service failing because of an existing bug in Kafka (https://issues.apache.org/jira/browse/KAFKA-1194) is a recognised problem.
The solution is to shutdown the Pega server and remove the Kafka data folder located at tomcat\kafka-data.
When you restart the server this directory will be recreated.
I will check internally to see if there are any plans to change the PE kafka provision.
Pegasystems Inc.
GB
@SatishKumarL5551 You stated that you were using PE 8.5.1 then it won't contain the fixes as it’s just a download of the Pega software pushed out by release engineering at GA time. .
The latest version of Kafka shipped with the known Kafka-on-Windows fixes is Kafka- Proprietary information hidden, a version that is available in the following patch releases:
Platform version | 'Kafka- Proprietary information hidden' availability |
---|---|
8.5.3+ | yes |
8.6+ | yes |
I am unable to comment when the PE version will be updated from 8.5.1 to 8.5.3
Please can you test by installing the latest version from here?
Areteans
NZ
@MarijeSchillern Thanks for the details. Let me check how I can replace current version and install Kafka- Proprietary information hidden as current version came along with PE instance. Please do share in case of any instructions around it.
Accepted Solution
Updated: 17 Feb 2022 23:44 EST
Pegasystems Inc.
GB
The below workaround was something I found which may be used to stabilize the affected environment, but cannot be considered as a permanent fix (and I don't have access to the jar files listed).
- When the platform starts it unpacks Kafka to the
kafka-1.1.0.X
folder, and then spins the Kafka process using the libraries from that folder. - For the following procedure the
kafka-1.1.0.X
folder should already exist on the disk.- Move out from the kafka-1.1.0.X/lib folder two jars:
kafka_2.11- Proprietary information hidden.jar
andkafka-clients- Proprietary information hidden.jar
- Copy into the kafka-1.1.0.X/lib folder two patched jars
kafka_2.11- Proprietary information hidden.jar
andkafka-clients- Proprietary information hidden.jar
from thekafka- Proprietary information hidden
distribution.
- Move out from the kafka-1.1.0.X/lib folder two jars:
- Restart Stream service or the node.
Areteans
NZ
@MarijeSchillern I was able to get the suggested jars and follow the steps suggested above which worked like a charm.
Appreciate your prompt responses in resolving my issue.
-
Marije Schillern