Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 PRADEEP CHOWDARY PYDI (PradeepPydi)
Anthem
Senior PEGA DEVOPS and Cloud Administrator
Anthem
US
PradeepPydi Member since 2017 240 posts
Anthem
Posted: Jul 13, 2018
Last activity: Mar 14, 2019
Posted: 13 Jul 2018 14:04 EDT
Last activity: 14 Mar 2019 4:47 EDT
Closed

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Hi Team,

We are unable to deploy the code to one of our environments because of the below mentioned errors.

PFA for the complete log.

Show More

Hi Team,

We are unable to deploy the code to one of our environments because of the below mentioned errors.

PFA for the complete log.

build	13-Jul-2018 10:57:06	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 10:57:06	     [java] 	at com.hazelcast.logging.LoggingServiceImpl$DefaultLogger.log(LoggingServiceImpl.java:166)
build	13-Jul-2018 10:57:06	     [java] 	at com.hazelcast.logging.LoggingServiceImpl$DefaultLogger.log(LoggingServiceImpl.java:158)
build	13-Jul-2018 10:57:06	     [java] 	at com.hazelcast.logging.AbstractLogger.finest(AbstractLogger.java:30)
build	13-Jul-2018 10:57:08	     [java] 	at com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor$MonitorInvocationsTask.run0(InvocationMonitor.java:286)
build	13-Jul-2018 10:57:08	     [java] 	at com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor$FixedRateMonitorTask.run(InvocationMonitor.java:257)
build	13-Jul-2018 10:57:08	     [java] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
build	13-Jul-2018 10:57:08	     [java] 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
build	13-Jul-2018 10:57:08	     [java] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
build	13-Jul-2018 10:57:08	     [java] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
build	13-Jul-2018 10:58:32	     [java] Exception in thread "hz._hzInstance_1_65b6da5135c3ecccb0f77ce104466df8.SlowOperationDetectorThread" java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 10:59:10	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 10:59:10	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 10:59:43	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:01:03	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:01:03	     [java] AsyncLogger error handling event seq=1468, value='Logger=com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor Level=FATAL Message=[ Proprietary information hidden]:5701 [65b6da5135c3ecccb0f77ce104466df8] [3.8] GC overhead limit exceeded':
build	13-Jul-2018 11:01:08	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:02:00	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:02	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] 2018-07-13 10:58:28,069 [        VDAASW702048] [  STANDARD] [                    ] [                    ] (pboard.StorageStreamCommonImpl) INFO    - [Encoding - Error during encode - 
build	13-Jul-2018 11:03:19	     [java] 2018-07-13 10:58:28,069 [        VDAASW702048] [  STANDARD] [                    ] [                    ] (pboard.StorageStreamCommonImpl) INFO    - HISTORY-RULE RULE-PORTALSKIN ANTMCSFW #20180605T210813.637 GMT!20180713T145527.074 GMT
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.nio.Packet.readValue(Packet.java:281)
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.nio.Packet.readFrom(Packet.java:274)
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.nio.tcp.MemberReadHandler.onRead(MemberReadHandler.java:58)
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.internal.networking.nonblocking.NonBlockingSocketReader.handle(NonBlockingSocketReader.java:161)
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKey(NonBlockingIOThread.java:350)
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKeys(NonBlockingIOThread.java:335)
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:253)
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:206)
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] Exception in thread "hz._hzInstance_1_65b6da5135c3ecccb0f77ce104466df8.IOBalancerThread" java.lang.RuntimeException: Unable to create engine instance - class path be wrong, or contain invalid contents
build	13-Jul-2018 11:03:19	     [java] 	at com.pega.pegarules.pub.PegaRULES.getEngine(PegaRULES.java:60)
build	13-Jul-2018 11:03:19	     [java] 	at com.pega.pegarules.priv.alert.AlertBuilder.ensureNodeSet(AlertBuilder.java:266)
build	13-Jul-2018 11:03:19	     [java] 	at com.pega.pegarules.priv.alert.AlertBuilder.withoutRequestor(AlertBuilder.java:127)
build	13-Jul-2018 11:03:19	     [java] 	at com.pega.pegarules.priv.LogHelper.alert(LogHelper.java:1768)
build	13-Jul-2018 11:03:19	     [java] 	at com.pega.pegarules.priv.AlertEvent$1.log(AlertEvent.java:82)
build	13-Jul-2018 11:03:19	     [java] 	at com.pega.pegarules.priv.LogHelper.doAlertException(LogHelper.java:1342)
build	13-Jul-2018 11:03:19	     [java] 	at com.pega.pegarules.priv.LogHelper.fatal(LogHelper.java:1312)
build	13-Jul-2018 11:03:19	     [java] 	at com.pega.platform.cluster.internal.util.HazelCastLogHelper.severe(HazelCastLogHelper.java:230)
build	13-Jul-2018 11:03:19	     [java] 	at com.pega.platform.cluster.internal.util.HazelCastLogHelper.log(HazelCastLogHelper.java:196)
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.logging.LoggingServiceImpl$DefaultLogger.log(LoggingServiceImpl.java:168)
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.logging.AbstractLogger.severe(AbstractLogger.java:75)
build	13-Jul-2018 11:03:19	     [java] 	at com.hazelcast.internal.networking.nonblocking.iobalancer.IOBalancerThread.run(IOBalancerThread.java:60)
build	13-Jul-2018 11:03:19	     [java] Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] Exception in thread "PegaRULES LicenseDaemon" java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] Exception in thread "PegaRULES UsageDaemon" java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
build	13-Jul-2018 11:03:19	     [java] AsyncLogger error handling event seq=1469, value='Logger=com.hazelcast.internal.networking.nonblocking.iobalancer.IOBalancer Level=FATAL Message=[ Proprietary information hidden]:5701 [65b6da5135c3ecccb0f77ce104466df8] [3.8] IOBalancer failed':
build	13-Jul-2018 11:03:19	     [java] java.lang.OutOfMemoryError: GC overhead limit exceeded

Show Less
To see attachments, please log in.
Low-Code App Development
System Administration

Posted: 6 years ago
Posted: 15 Jul 2018 22:07 EDT
Harish Gunneri (Harish_Gunneri)
PEGA
Senior Principal Engineer, Technical Support
Pegasystems Inc.
US

Hi Pradeep,

 

It looks like the deployment of the archive zip file failed on the target Pega DB due to java.lang.OutOfMemoryError: GC overhead limit exceeded

This message means that for some reason the garbage collector is taking an excessive amount of time (by default 98% of all CPU time of the process) and recovers very little memory in each run (by default 2% of the heap). 

This error is to notify server admins (to take corrective actions)  to prevent applications from running for an extended period of time while making little or no progress because the heap is too small.  

 

For deployment using PRPCUtils embedded server initialization will be done followed by rap import. It looks like the max heap is specified as 1 gig for this process.

As per the logs: JVM Arguments -XX:MaxHeapSize=1280m

Try increasing it to higher heap size and retry the deployment. Let me know how it goes.

Hope this helps!

To see attachments, please log in.
Posted: 6 years ago
Posted: 15 Jul 2018 23:45 EDT
PRADEEP CHOWDARY PYDI (PradeepPydi)
Anthem
Senior PEGA DEVOPS and Cloud Administrator
Anthem
US

Hi Harish,

Strangely,I dont have MaxHeapSize argument in the JVM's argument list.

PFB for the list of arguments.

-Dcatalina.home=xxx
-Dcatalina.base=xxx
-Djava.endorsed.dirs=xxx
-Djava.io.tmpdir=xxx
-Djavax.net.ssl.trustStoreType=xxx
-Djavax.net.ssl.trustStore=xxx
-Djavax.net.ssl.trustStorePassword=xxx
-Djsse.enableSNIExtension=false
-XX:HeapDumpPath=xxx
-Xloggc:xxx
-Xms8192m
-Xmx8192m
-XX:+UseG1GC
-XX:MaxGCPauseMillis=350
-XX:InitiatingHeapOccupancyPercent=45
-XX:ReservedCodeCacheSize=512m
-XX:+PrintAdaptiveSizePolicy
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+DisableExplicitGC
-XX:MetaspaceSize=512m
-XX:MaxMetaspaceSize=2048m
-Djava.awt.headless=true
-verbose:gc
-XX:+HeapDumpOnOutOfMemoryError
-XX:+PrintHeapAtGC
-XX:+ParallelRefProcEnabled

PFA for the latest log.

To see attachments, please log in.
Posted: 6 years ago
Posted: 16 Jul 2018 1:59 EDT
Pawan Narain Singh (pawann)
Swedbank AB

Swedbank AB
SE

Hello Pradeep,

Xmx8192m denotes that you have 8192MB of maximum Java Heap Size. I believe that "prpcUtilsTarget_Compass_CI.properties" is setting "XX:MaxHeapSize" to "1280m" and is overriding your JVM argument list.

Try locating and removing this setting from the file to resolve this.

 

Thank you,
Pawan

To see attachments, please log in.
Posted: 5 years ago
Posted: 13 Mar 2019 11:28 EDT
Ankur Maheshwari (AnkurM16)
JP Morgan

JP Morgan
IN

hi,

 

Please check the prpcUtil.xml file. Pega has set the max limit. We can modify the same to our need.

Show More

hi,

 

Please check the prpcUtil.xml file. Pega has set the max limit. We can modify the same to our need.

Extract from the file-
 
<ac:then>
                   <condition property="jvm.arg.mem.max.temp" value="${jvm.arg.mem.max}" else="-Xmx1280m">
                       <isset property="jvm.arg.mem.max"/>
                   </condition>
                   <condition property="jvm.arg.mem.maxpermsize.temp" value="${jvm.arg.mem.maxpermsize}" else="-XX:MaxPermSize=512m">
                       <isset property="jvm.arg.mem.maxpermsize"/>
                   </condition>
                   <property name="jvm.args.temp" value="${jvm.arg.mem.max.temp} ${jvm.arg.mem.maxpermsize.temp}"/>
               </ac:then>
               <ac:else>
                   <condition property="jvm.arg.mem.max.temp" value="${jvm.arg.mem.max}" else="-XX:MaxHeapSize=1280m">
                       <isset property="jvm.arg.mem.max"/>
                   </condition>
                   <property name="jvm.args.temp" value="${jvm.arg.mem.max.temp}"/>
 
Thanks

Show Less
To see attachments, please log in.
Posted: 5 years ago
Posted: 14 Mar 2019 4:47 EDT
Simon Chan (chans3)
PEGA
Senior Principal Engineer, Technical Support - 1:1 Customer Engagement
Pegasystems Inc.
GB

You could limit the GC time by including the JVM argument of -XX:+UseGCOverheadLimit.

In addition, I would also include a GC policy option in the JVM argument.

Reference:
https://www.oracle.com/technetwork/articles/java/vmoptions-jsp-140102.html

Hope it helps

Simon

To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice