Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 Faisal Khan (Faisal Khan)
Inovar

Inovar
IN
Faisal Khan Member since 2021 3 posts
Inovar
Posted: Aug 19, 2019
Last activity: Aug 22, 2019
Posted: 19 Aug 2019 13:00 EDT
Last activity: 22 Aug 2019 10:50 EDT
Closed

Hazelcast Issues Pega 8

We are observing a lot of issues related to hazelcast in our environment.

Sometime hazelcast engine goes down still the prsysstatus node tables gets updated. PegaCluster log stops getting updated.

2019-08-17 04:25:01,920 [331.cached.thread-46] [ ] [ ] ( hazelcast.instance.Node) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Shutting down node engine...

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] (zelcast.instance.NodeExtension) INFO - [ Proprietary information hidden]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Destroying node NodeExtension.

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] ( hazelcast.instance.Node) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Hazelcast Shutdown is completed in 106401 ms.

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] [xxxxx]:5706 is SHUTDOWN

2019-08-17 04:25:05,321 [331.cached.thread-46] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] [xxxxx]:5706 is MERGE_FAILED

We get Pega0090 alerts.

A lot of times there are communication failures between members.

Show More

We are observing a lot of issues related to hazelcast in our environment.

Sometime hazelcast engine goes down still the prsysstatus node tables gets updated. PegaCluster log stops getting updated.

2019-08-17 04:25:01,920 [331.cached.thread-46] [ ] [ ] ( hazelcast.instance.Node) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Shutting down node engine...

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] (zelcast.instance.NodeExtension) INFO - [ Proprietary information hidden]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Destroying node NodeExtension.

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] ( hazelcast.instance.Node) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Hazelcast Shutdown is completed in 106401 ms.

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] [xxxxx]:5706 is SHUTDOWN

2019-08-17 04:25:05,321 [331.cached.thread-46] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] [xxxxx]:5706 is MERGE_FAILED

We get Pega0090 alerts.

A lot of times there are communication failures between members.

Reason: Exception in Connection[id=602, /yyyyy:5707->/yyyyy:55601, endpoint=[yyyyyy]:5706, alive=true, type=MEMBER], thread=hz._hzInstance_1_a948064bf1085f5cb78ac1448bc1f331.IO.thread-in-1][STACK][java.io.IOException: Connection reset by peer<CR> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)<CR> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)<CR> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)<CR> at sun.nio.ch.IOUtil.read(IOUtil.java:197)<CR> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)<CR> at com.hazelcast.nio.ssl.SSLChannel.read(SSLChannel.java:189)<CR> at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:129)<CR> at com.hazelcast.internal.networking.nio.NioThread.handleSelectionKey(NioThread.java:383)<CR> at com.hazelcast.internal.networking.nio.NioThread.handleSelectionKeys(NioThread.java:368)<CR> at com.hazelcast.internal.networking.nio.NioThread.selectLoopWithFix(NioThread.java:289)<CR> at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:224)

Sub clusters getting formed

2019-08-17 04:23:54,941 [448bc1f331.migration] [ ] [ ] (artition.impl.MigrationManager) INFO - [yyyyyy]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Partition balance is ok, no need to re-partition cluster data...
2019-08-17 04:23:57,935 [331.cached.thread-17] [ ] [ ] (nternal.cluster.ClusterService) INFO - [yyyyyy]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10]
Members {size:1, ver:201} [
Member [yyyyy]:5711 - 54c532a9-36c3-4968-91c7-ae591b16648c this
]
2019-08-17 04:23:57,935 [331.cached.thread-17] [ ] [ ] (cluster.impl.MembershipManager) INFO - [xxxxxx]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Mastership is claimed with: MembersView{version=201, members=[MemberInfo{address=[xxxxxxx]:5711, uuid=54c532a9-36c3-4968-91c7-ae591b16648c, liteMember=false, memberListJoinVersion=200}]}
Anyone has a cluster size of around 60?

***Edited by Moderator Marissa to update platform capability tags; update SR Details****


Show Less
To see attachments, please log in.
Pega Platform
System Administration
Support Case Exists

Posted: 5 years ago
Posted: 19 Aug 2019 17:26 EDT
Sreecharan Babu Challagundla (SREECHARAN)
Cognizant Technologies Solutions US
Sreecharan Babu Challagundla
Cognizant Technologies Solutions US
CA

Hello Faisal,

It is better if you can create custom hazelcast groups... another reg: port communication, seems b/w servers ports are not opened properly.

Thanks

 

To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice