Hello Team,

We have a huge problem every time the prod server stops with a 504 error and when I chek the log I find:

Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: State: SHUT_DOWN Operation: class com.hazelcast.map.impl.query.QueryPartitionOperation

I don't know exactly what does it means.

I will appreciate it if anyone can help what should I do in that case?

Thank you

Best Regards,

Imen

2021-10-07 13:40:36,752 [   PegaRULES-Batch-4] [  STANDARD] [                    ] [   CustomerService:8] (ternal.HazelcastDistributedMap) ERROR   - DMAP [default#com.pega.pegarules.session.internal.async.AgentInfo] Operation [get(aKey)] failed on key [GENERATEACCESSFOROPERATOR:PEGACRM-SFA:PEGA-NODE01] com.pega.pegarules.pub.PRRuntimeException: PRRuntimeException     at com.pega.platform.hazelcast.util.HazelcastOperation.perform(HazelcastOperation.java:69) ~[hazelcast.jar:?]     at com.pega.platform.datagridhz.map.internal.HazelcastDistributedMap.executeWithRetry(HazelcastDistributedMap.java:295) ~[datagrid-hz.jar:?]     at com.pega.platform.datagridhz.map.internal.HazelcastDistributedMap.get(HazelcastDistributedMap.java:100) ~[datagrid-hz.jar:?]     at com.pega.platform.datagrid.map.internal.ProxyDistributedMap.get(ProxyDistributedMap.java:81) ~[datagrid.jar:?]     at com.pega.pegarules.session.internal.async.AgentInfoManager.getAgentInfo(AgentInfoManager.java:71) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.async.AgentQueue.updateAgentInfo(AgentQueue.java:979) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.async.AgentQueue.updateAgentInfo(AgentQueue.java:974) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.async.AgentQueue.runEnd(AgentQueue.java:1135) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.async.AgentQueue.runEnd(AgentQueue.java:1085) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:1214) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1397) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1140) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1021) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:818) ~[prprivate-session.jar:?]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_272]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_272]     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272] Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!     at com.hazelcast.instance.EnterpriseNodeExtension$1.get(EnterpriseNodeExtension.java:378) ~[hazelcast-enterprise-3.11.5.jar:3.11.5]     at com.hazelcast.instance.EnterpriseNodeExtension$1.get(EnterpriseNodeExtension.java:375) ~[hazelcast-enterprise-3.11.5.jar:3.11.5]     at com.hazelcast.internal.serialization.impl.bufferpool.BufferPoolThreadLocal.get(BufferPoolThreadLocal.java:107) ~[hazelcast-enterprise-3.11.5.jar:3.11.5]     at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:141) ~[hazelcast-enterprise-3.11.5.jar:3.11.5]     at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:133) ~[hazelcast-enterprise-3.11.5.jar:3.11.5]     at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:118) ~[hazelcast-enterprise-3.11.5.jar:3.11.5]     at com.hazelcast.map.impl.proxy.MapProxySupport.toDataWithStrategy(MapProxySupport.java:1262) ~[hazelcast-enterprise-3.11.5.jar:3.11.5]     at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:342) ~[hazelcast-enterprise-3.11.5.jar:3.11.5]     at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:120) ~[hazelcast-enterprise-3.11.5.jar:3.11.5]     at com.pega.platform.datagridhz.map.internal.HazelcastDistributedMap$2.call(HazelcastDistributedMap.java:103) ~[datagrid-hz.jar:?]     at com.pega.platform.datagridhz.map.internal.HazelcastDistributedMap$2.call(HazelcastDistributedMap.java:100) ~[datagrid-hz.jar:?]     at com.pega.platform.datagridhz.map.internal.HazelcastDistributedMap.lambda$executeWithRetry$6(HazelcastDistributedMap.java:295) ~[datagrid-hz.jar:?]     at com.pega.platform.hazelcast.util.HazelcastOperation.perform(HazelcastOperation.java:52) ~[hazelcast.jar:?]     ... 16 more 2021-10-07 13:40:41,342 [onStateTrackerDaemon] [          ] [                    ] [                    ] (.PresenceLifeCycleEventManager) ERROR   - Failed to update session state for requestor with ID: HKGNEZWWZX0EI8JRRJMIC3U33FM24KML2A com.pega.pegarules.pub.PRRuntimeException: PRRuntimeException     at com.pega.platform.hazelcast.util.HazelcastOperation.perform(HazelcastOperation.java:69) ~[hazelcast.jar:?]     at com.pega.pegarules.cluster.internal.presence.holders.PresenceDataGridHazelcast.query(PresenceDataGridHazelcast.java:172) ~[prcluster.jar:?]     at com.pega.pegarules.cluster.internal.presence.utils.PresenceQueryHelper.getPartPrenceRecordList(PresenceQueryHelper.java:111) ~[prcluster.jar:?]     at com.pega.pegarules.cluster.internal.presence.utils.PresenceQueryHelper.queryPresenceBatchWiseAndReturnMergedList(PresenceQueryHelper.java:65) ~[prcluster.jar:?]     at com.pega.pegarules.cluster.internal.presence.holders.PresenceDataGridHazelcast.getPresenceRecordsByRequestorIds(PresenceDataGridHazelcast.java:312) ~[prcluster.jar:?]     at com.pega.pegarules.session.internal.presence.PresenceDataProcessorImpl.getPresenceRecordInternals(PresenceDataProcessorImpl.java:338) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.presence.PresenceDataProcessorImpl.updateSessionStateToDisconnected(PresenceDataProcessorImpl.java:308) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.presence.PresenceLifeCycleEventManager.updateSessiontoDisconnectedIfIdle(PresenceLifeCycleEventManager.java:162) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.presence.PresenceSessionStateTrackerDaemon.identifyDisconnectedSessions(PresenceSessionStateTrackerDaemon.java:162) ~[prprivate-session.jar:?]     at com.pega.pegarules.session.internal.presence.PresenceSessionStateTrackerDaemon.run(PresenceSessionStateTrackerDaemon.java:132) ~[prprivate-session.jar:?]     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272] Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: State: SHUT_DOWN Operation: class com.hazelcast.map.impl.query.QueryPartitionOperation