Question
Anthem
US
Last activity: 7 Feb 2019 15:34 EST
FATAL errors @ PegaCLUSTER logs @ PRPC731 version.
Hi Team,
We could find a lot of FATAL errors in the PegaCLUSTER logs.
Please suggest.Also find the attachment for complete log.
1)2019-02-06 22:05:06,320 [c-operation.thread-0] [ ] [ ] (.operations.HeartbeatOperation) FATAL - [xx.xx.xx.xx]:5707 [65b6da5135c3ecccb0f77ce104466df8] [3.8] Java heap space
java.lang.OutOfMemoryError: Java heap space
2)2019-02-06 22:12:27,853 [ocationMonitorThread] [ ] [ ] (service.impl.InvocationMonitor) FATAL - [xx.xx.xx.xx]java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3a3c927c rejected from java.util.concurrent.ScheduledThreadPoolExecutor@3f9bd684[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 808]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3a3c927c rejected from java.util.concurrent.ScheduledThreadPoolExecutor@3f9bd684[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 808]
Hi Team,
We could find a lot of FATAL errors in the PegaCLUSTER logs.
Please suggest.Also find the attachment for complete log.
1)2019-02-06 22:05:06,320 [c-operation.thread-0] [ ] [ ] (.operations.HeartbeatOperation) FATAL - [xx.xx.xx.xx]:5707 [65b6da5135c3ecccb0f77ce104466df8] [3.8] Java heap space
java.lang.OutOfMemoryError: Java heap space
2)2019-02-06 22:12:27,853 [ocationMonitorThread] [ ] [ ] (service.impl.InvocationMonitor) FATAL - [xx.xx.xx.xx]java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3a3c927c rejected from java.util.concurrent.ScheduledThreadPoolExecutor@3f9bd684[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 808]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3a3c927c rejected from java.util.concurrent.ScheduledThreadPoolExecutor@3f9bd684[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 808]
3)2019-02-06 22:12:48,514 [ocationMonitorThread] [ ] [ ] (service.impl.InvocationMonitor) FATAL - [xx.xx.xx.xx]:5707 [65b6da5135c3ecccb0f77ce104466df8] [3.8] Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1d22d7c9 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@3f9bd684[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 808]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1d22d7c9 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@3f9bd684[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 808]
4)2019-02-06 22:15:22,233 [e104466df8.scheduled] [ ] [ ] (hazelcast.spi.ExecutionService) FATAL - [xx.xx.xx.xx]:5707 [65b6da5135c3ecccb0f77ce104466df8] [3.8] Failed to execute LoggingDelegatingFuture{task=DelegateAndSkipOnConcurrentExecutionDecorator{isAlreadyRunning=true, runnable=com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator$DelegateDecorator@2de52086, executor=com.hazelcast.util.executor.CachedExecutorServiceDelegate@6c6df1fc}}
java.util.concurrent.RejectedExecutionException
Pegasystems Inc.
US
It looks like there was a heap space issue, you may need to increase the heap size.
Can the nodes communicate with each other across the hazelcast ports? (5701-5709 in your case it looks like)
Do you have multiple Network Interface Cards on the server hosting the nodes? If so, you may need to specify the IP that should be used for clustering in the prconfig.xml file using the following setting:
<env name="cluster/hazelcast/interface" value="xx.xx.xx.xx"/>