Exception in the Kafka server.log

[2021-11-09 13:06:11,084] ERROR Failed to clean up log for df_run_metrics-4 in dir

D:\opt\PEGA\ldn\test\temp\kafka-data due to IOException (kafka.server.LogDirFailureChannel)

java.nio.channels.ClosedChannelException: null

at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:122) ~[?:1.8.0]

at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:737) ~[?:1.8.0]

at org.apache.kafka.common.utils.Utils.readFully(Utils.java:831) ~[kafka-clients-1.1.0.5.jar:?]

at org.apache.kafka.common.record.FileRecords.readInto(FileRecords.java:116) ~[kafka-clients-1.1.0.5.jar:?]

at kafka.log.Cleaner.cleanInto(LogCleaner.scala:601) ~[kafka_2.11-1.1.0.5.jar:?]

at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:518) ~[kafka_2.11-1.1.0.5.jar:?]

Solution

Prior to Pega Platform version 8.6, the Pega-provided Kafka distribution uses two log compacted topics: __consumer_offsets, and df_run_metrics.

To resolve the Kafka server.log exception, convert these three topics to uncompacted topics by specifying the Kafka cleanup policy as delete:

$.\kafka-1.1.0.5\bin\windows\kafka-topics.bat --zookeeper localhost:2181 --topic __consumer_offsets --alter --config cleanup.policy=delete

$.\kafka-1.1.0.5\bin\windows\kafka-topics.bat --zookeeper localhost:2181 --topic df_run_metrics --alter --config cleanup.policy=delete

$.\kafka-1.1.0.5\bin\windows\kafka-topics.bat --zookeeper localhost:2181 --topic stream_dataset_offsets --alter --config cleanup.policy=delete

Related content

Third-party documentation

IBM WebSphere Application Server v8.5.5 Fix Pack 19

Kafka Log Compaction

Kafka Topic-Level Configs See cleanup.policy.

Pega Documentation

Kafka as a streaming service

Best practices for Stream service configuration

Troubleshooting Kafka