Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 Lanh NguyenTrung (LANH（ラン）) Toyota System Pega Development Team Lead
Toyota System
JP
LANH（ラン） Member since 2018 15 posts
Toyota System
Posted: Jan 28, 2021
Last activity: Apr 8, 2021
Posted: 28 Jan 2021 1:34 EST
Last activity: 8 Apr 2021 6:15 EDT
Closed

Kafka shutting down

Our system usually shutting down kafka stream service. I had checked server.log and know that when client session time out, it reconnect and then shutting down

1-01-27 19:40:15,726] WARN Client session timed out, have not heard from server in 20002ms for sessionid 0xffffffffbcf60001 (org.apache.zookeeper.ClientCnxn)

[2021-01-27 19:40:15,726] INFO Client session timed out, have not heard from server in 20002ms for sessionid 0xffffffffbcf60001, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

[2021-01-27 19:40:17,653] INFO Opening socket connection to server  Proprietary information hidden/ Proprietary information hidden:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2021-01-27 19:40:17,653] INFO Socket connection established to  Proprietary information hidden/ Proprietary information hidden:2181, initiating session (org.apache.zookeeper.ClientCnxn)

[2021-01-27 19:40:17,654] INFO Session establishment complete on server  Proprietary information hidden/ Proprietary information hidden:2181, sessionid = 0xffffffffbcf60001, negotiated timeout = 30000 (org.apache.zookeeper.ClientCnxn)

Show More

Our system usually shutting down kafka stream service. I had checked server.log and know that when client session time out, it reconnect and then shutting down

1-01-27 19:40:15,726] WARN Client session timed out, have not heard from server in 20002ms for sessionid 0xffffffffbcf60001 (org.apache.zookeeper.ClientCnxn)

[2021-01-27 19:40:15,726] INFO Client session timed out, have not heard from server in 20002ms for sessionid 0xffffffffbcf60001, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

[2021-01-27 19:40:17,653] INFO Opening socket connection to server  Proprietary information hidden/ Proprietary information hidden:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2021-01-27 19:40:17,653] INFO Socket connection established to  Proprietary information hidden/ Proprietary information hidden:2181, initiating session (org.apache.zookeeper.ClientCnxn)

[2021-01-27 19:40:17,654] INFO Session establishment complete on server  Proprietary information hidden/ Proprietary information hidden:2181, sessionid = 0xffffffffbcf60001, negotiated timeout = 30000 (org.apache.zookeeper.ClientCnxn)

[2021-01-27 19:40:52,505] INFO Terminating process due to signal SIGTERM (org.apache.kafka.common.utils.LoggingSignalHandler)

[2021-01-27 19:40:52,535] INFO [KafkaServer id=1003] shutting down (kafka.server.KafkaServer)

[2021-01-27 19:40:52,536] INFO [KafkaServer id=1003] Starting controlled shutdown (kafka.server.KafkaServer)

[2021-01-27 19:40:52,611] INFO [KafkaServer id=1003] Controlled shutdown succeeded (kafka.server.KafkaServer)

Show Less
***Edited by Moderator: Pooja Gadige to add platform capability tag***
To see attachments, please log in.
Pega Platform 8.4.2 Decision Management System Administration System Architect

  • Tarun Sipani
Posted: 3 years ago
Posted: 25 Feb 2021 3:47 EST
Tarun Sipani (sipat) Senior Technical Support Engineer
Pegasystems Inc.
JP

Hi ,

This excerpt of the logs is from the Kafka-server-logs itself. We are still unable to figure out why this error is seen when the Stream-node is re-booted . 

Any help will be much appreciated. 

Regards,

To see attachments, please log in.
Posted: 3 years ago
Posted: 8 Apr 2021 6:15 EDT
Abikash Kumar Sharma (AbikashSharma) Capgemini Abikash Kumar Sharma
Capgemini
SE

@sipat I have a similar problem in our dev environment. It has only one which acts as both web and stream node. The stream node suddenly becomes unreachable after some days. I could see the following in kafka server logs

Probably because of GC it is crossing the 20008ms threshold. I have seen a blog where it is recommended to either tune the GC or increase zookeeper.session.timeout.ms in the broker config. Is it possible with some DSS in Pega?

https://stackoverflow.com/questions/38859944/maximum-value-for-zookeeper-connection-timeout-ms

Any inputs is highly appreciated.

Show More

@sipat I have a similar problem in our dev environment. It has only one which acts as both web and stream node. The stream node suddenly becomes unreachable after some days. I could see the following in kafka server logs

Probably because of GC it is crossing the 20008ms threshold. I have seen a blog where it is recommended to either tune the GC or increase zookeeper.session.timeout.ms in the broker config. Is it possible with some DSS in Pega?

https://stackoverflow.com/questions/38859944/maximum-value-for-zookeeper-connection-timeout-ms

Any inputs is highly appreciated.

[2021-04-08 10:00:31,827] WARN Client session timed out, have not heard from server in 20008ms for sessionid 0x7887f23f2e0001 (org.apache.zookeeper.ClientCnxn)
[2021-04-08 10:00:31,827] INFO Client session timed out, have not heard from server in 20008ms for sessionid 0x7887f23f2e0001, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2021-04-08 10:01:17,887] INFO Opening socket connection to server  Proprietary information hidden/ Proprietary information hidden:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-04-08 10:01:17,902] INFO Socket connection established to  Proprietary information hidden/ Proprietary information hidden:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2021-04-08 10:01:44,191] WARN Unable to reconnect to ZooKeeper service, session 0x7887f23f2e0001 has expired (org.apache.zookeeper.ClientCnxn)
[2021-04-08 10:01:44,191] INFO Unable to reconnect to ZooKeeper service, session 0x7887f23f2e0001 has expired, closing socket connection (org.apache.zookeeper.ClientCnxn)
[2021-04-08 10:01:44,318] INFO EventThread shut down for session: 0x7887f23f2e0001 (org.apache.zookeeper.ClientCnxn)
[2021-04-08 10:01:44,415] INFO [ZooKeeperClient] Session expired. (kafka.zookeeper.ZooKeeperClient)
[2021-04-08 10:01:44,515] INFO [ZooKeeperClient] Initializing a new session to  Proprietary information hidden:2181. (kafka.zookeeper.ZooKeeperClient)
[2021-04-08 10:01:44,516] INFO Initiating client connection, connectString= Proprietary information hidden:2181 sessionTimeout=30000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@7d3e8655 (org.apache.zookeeper.ZooKeeper)
[2021-04-08 10:01:44,716] INFO Opening socket connection to server  Proprietary information hidden/ Proprietary information hidden:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-04-08 10:01:44,717] INFO Socket connection established to  Proprietary information hidden/ Proprietary information hidden:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2021-04-08 10:01:44,871] INFO Session establishment complete on server  Proprietary information hidden/ Proprietary information hidden:2181, sessionid = 0x7887f23f2e0002, negotiated timeout = 30000 (org.apache.zookeeper.ClientCnxn)
[2021-04-08 10:01:45,525] INFO Creating /brokers/ids/1 (is it secure? false) (kafka.zk.KafkaZkClient)
[2021-04-08 10:01:45,799] INFO Result of znode creation at /brokers/ids/1 is: OK (kafka.zk.KafkaZkClient)
[2021-04-08 10:01:45,799] INFO Registered broker 1 at path /brokers/ids/1 with addresses: ArrayBuffer(EndPoint( Proprietary information hidden,9092,ListenerName(PLAINTEXT),PLAINTEXT)) (kafka.zk.KafkaZkClient)
[2021-04-08 10:01:45,970] INFO Creating /controller (is it secure? false) (kafka.zk.KafkaZkClient)
[2021-04-08 10:01:46,100] INFO Result of znode creation at /controller is: OK (kafka.zk.KafkaZkClient)

 

Show Less
To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice