Kafka log messages piling up
The tomcat std err file keeps growing quickly because of the kafka logs.
Below is a snippet from the log file. This message keeps on getting logged repetitively, until the disk space runs out.
Any suggestions will help us identifying the cause here.
The tomcat std err file keeps growing quickly because of the kafka logs.
Below is a snippet from the log file. This message keeps on getting logged repetitively, until the disk space runs out.
Any suggestions will help us identifying the cause here.
[pool-5728-thread-1] INFO org.apache.kafka.clients.admin.AdminClientConfig - AdminClientConfig values:
bootstrap.servers = [ Proprietary information hidden:9092]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 120000
retries = 5
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
HCA Healthcare
US
@NisargK0The repetitive Kafka log messages in the Tomcat stderr file may be due to configuration or integration issues. First, analyze the log messages to identify errors like broker connectivity, topic misconfiguration, or authentication issues. Check both Kafka and application logs for more details and adjust the Kafka logging level to ERROR in the configuration file to reduce unnecessary logs. Ensure the Kafka client library matches the broker version and verify all brokers are reachable. Additionally, set up log rotation for the Tomcat stderr file to prevent disk space exhaustion. Free up space temporarily by archiving old logs and use monitoring tools to keep track of Kafka’s health. These steps should help resolve and prevent the issue
Ciberspring Inc
IN
@SiddhantJivane No. We have an SR request open and Pega is trying to identify the root cause.
Ciberspring Inc
IN
Hey I remember seeing a comment here with some db queries truncating stream tables, but now can't find it now.
Areteans
AU
@NisargK0, We truncated kafka logs and kafka tables - servers were up and running for couple of days but again we faced the issue of kafka logs piling up until server runs out of disk space - we have raised incident with Pega as well - our team is waiting for resolution.
Ciberspring Inc
IN
We got the below response from Pega. Is this what you tried and did not work?
please perform the below truncates: Important: This procedure results in Kafka Data Loss. 1. Bring all Stream service instances down. 2. In the header of Dev Studio, click Configure > Decisioning > Infrastructure > Services > Stream. 3. Decommission all Stream service instances. 4. Truncate the Stream service tables in the Pega Platform data schema: - truncate table pr_data_stream_nodes - truncate table pr_data_stream_sessions - truncate table pr_data_stream_node_updates 5. Delete Kafka-data directory from the Stream instances. Note: This directory, by default is created under Tomcat folder. 6. Bring the Stream instances back up one instance at a time. Once done, wait for few minutes for the Stream nodes to sync the replicas and stabilize. Confirm that all Stream nodes are showing 0 for Offline and Under-replicated fields and there's only one node showing as Controller on Stream landing page.