Hello,

We are experiencing significant issues with Hazelcast where members are leaving the cluster and not rejoining until we perform a restart. This is causing considerable business impacts.

Details:

Product: Pega Proprietary information hidden Hazelcast Version: 5.2.4 (embedded)

Crosschecked other factors like Network, EC2 instances and none seems to have caused the problem.  

Symptoms:

Members leave the cluster unexpectedly. Members do not rejoin automatically; manual restart required. Error logs indicate issues such as TargetNotMemberException and IllegalStateException related to partition updates.  

Questions:

Are there any known bugs or issues with Hazelcast version 5.2.4 that could be causing these problems? Could the embedded version of Hazelcast be the root cause? What is the recommended process for upgrading to version 5.5.0? If we upgrade to 5.5.0, is there a possibility of encountering similar issues?  

Error Log Snippet:

[iendly_moser.event-1] [STANDARD] [ ] [ ] (til.HazelcastMembershipManager) INFO - Member has left the cluster: Member: [name=prod-express-app-2, address=CAVIPAPP0244.hewitt.com/ Proprietary information hidden:5701, uuid=8a026961-6df9-42d3-a4b5-43233a161622, member version=5.2.4, mode=SERVER]

[com.pega.hazelcast.v5.spi.exception.TargetNotMemberException: Not Member! target: [ Proprietary information hidden]:5702 - 6206ed34-5f42-41f2-bb37-c6239b95de2a, partitionId: 52, operation: com.pega.hazelcast.v5.map.impl.operation.PutOperation, service: hz:impl:mapService]

java.lang.IllegalStateException: Partition updates are diverged! Local: Partition {ID: 28, Version: 13} [0:[ Proprietary information hidden]:5702 - 5e716371-206c-4b4f-953e-89eb204df154 1:[ Proprietary information hidden]:5702 - 59596ebf-6e10-4f76-88fe-e49fb6395f1a 2:[ Proprietary information hidden]:5702 - 1378809b-4b10-4bdf-935c-aae15d76c3ae 3:[ Proprietary information hidden]:5702 - 3c816ede-5c82-4412-89db-7430aeffad89 4:[ Proprietary information hidden]:5702 - 6206ed34-5f42-41f2-bb37-c6239b95de2a 5:[ Proprietary information hidden]:5702 - 70cccf73-6579-48ef-82e7-74663c80e7a6 6:[ Proprietary information hidden]:5702 - 8a686614-8533-4f4c-8ac3-70015cc5b129], Received: Partition {ID: 28, Version: 13} [1:[ Proprietary information hidden]:5702 - 59596ebf-6e10-4f76-88fe-e49fb6395f1a 2:[ Proprietary information hidden]:5702 - 1378809b-4b10-4bdf-935c-aae15d76c3ae 3:[ Proprietary information hidden]:5702 - 3c816ede-5c82-4412-89db-7430aeffad89 4:[ Proprietary information hidden]:5702 - 6206ed34-5f42-41f2-bb37-c6239b95de2a 5:[ Proprietary information hidden]:5702 - 70cccf73-6579-48ef-82e7-74663c80e7a6 6:[ Proprietary information hidden]:5702 - 8a686614-8533-4f4c-8ac3-70015cc5b129] Any insights or recommendations would be greatly appreciated.

Thank you.