GC Allocation Failure after switching to EAR from WAR
We are trying to switch to ear from war on our Pega 718/JBoss EAP6.3 environment. I made the required changes and started the server. The prpc_j2ee14_jboss61JBM.ear gets deployed fine and I'm able to log in to the designer studio. But in the logs I see continuous messages about GC Allocation failure as below.
This was not occuring when I had the prweb.war deployed.
[GC (Allocation Failure) [ParNew: 1717206K->37541K(1887488K), 0.0673186 secs] 1983995K->304329K(16567552K), 0.0675282 secs] [Times: user=0.40 sys=0.00, real=0.06 secs]
[GC (Allocation Failure) [ParNew: 1715365K->36097K(1887488K), 0.0570058 secs] 1982153K->302936K(16567552K), 0.0572310 secs] [Times: user=0.37 sys=0.01, real=0.06 secs]
[GC (Allocation Failure) [ParNew: 1713921K->43639K(1887488K), 0.0562759 secs] 1980760K->310497K(16567552K), 0.0564710 secs] [Times: user=0.40 sys=0.00, real=0.05 secs]
[GC (Allocation Failure) [ParNew: 1721463K->27839K(1887488K), 0.0650930 secs] 1988321K->303971K(16567552K), 0.0653020 secs] [Times: user=0.42 sys=0.01, real=0.07 secs]
[GC (Allocation Failure) [ParNew: 1705663K->16370K(1887488K), 0.0691633 secs] 1981795K->302724K(16567552K), 0.0693636 secs] [Times: user=0.32 sys=0.02, real=0.07 secs]
[GC (Allocation Failure) [ParNew: 1694194K->5392K(1887488K), 0.0550236 secs] 1980548K->300214K(16567552K), 0.0552261 secs] [Times: user=0.36 sys=0.00, real=0.06 secs]
I checked the heap utilization is too low and there is almost 15GB of heap free.
We are trying to switch to ear from war on our Pega 718/JBoss EAP6.3 environment. I made the required changes and started the server. The prpc_j2ee14_jboss61JBM.ear gets deployed fine and I'm able to log in to the designer studio. But in the logs I see continuous messages about GC Allocation failure as below.
This was not occuring when I had the prweb.war deployed.
[GC (Allocation Failure) [ParNew: 1717206K->37541K(1887488K), 0.0673186 secs] 1983995K->304329K(16567552K), 0.0675282 secs] [Times: user=0.40 sys=0.00, real=0.06 secs]
[GC (Allocation Failure) [ParNew: 1715365K->36097K(1887488K), 0.0570058 secs] 1982153K->302936K(16567552K), 0.0572310 secs] [Times: user=0.37 sys=0.01, real=0.06 secs]
[GC (Allocation Failure) [ParNew: 1713921K->43639K(1887488K), 0.0562759 secs] 1980760K->310497K(16567552K), 0.0564710 secs] [Times: user=0.40 sys=0.00, real=0.05 secs]
[GC (Allocation Failure) [ParNew: 1721463K->27839K(1887488K), 0.0650930 secs] 1988321K->303971K(16567552K), 0.0653020 secs] [Times: user=0.42 sys=0.01, real=0.07 secs]
[GC (Allocation Failure) [ParNew: 1705663K->16370K(1887488K), 0.0691633 secs] 1981795K->302724K(16567552K), 0.0693636 secs] [Times: user=0.32 sys=0.02, real=0.07 secs]
[GC (Allocation Failure) [ParNew: 1694194K->5392K(1887488K), 0.0550236 secs] 1980548K->300214K(16567552K), 0.0552261 secs] [Times: user=0.36 sys=0.00, real=0.06 secs]
I checked the heap utilization is too low and there is almost 15GB of heap free.
Any tuning to be done on this for ear file?
Accepted Solution
Pegasystems Inc.
US
G1 is supposed to replace CMS collector eventually, since you are using java 1.8, you probably want to go to that route. All our current recommendation is based on field experience with java 6/7, so it will take some time for pega to actually collect enough data for java 1.8 systems, both internally and externally. For example, Redhat would recommend the following to start with - but ultimately jvm tuning should be really based on your own applications with sufficient load tests with realistic use case scenarios.
-server -XX:+DoEscapeAnalysis -XX:+UseCompressedOops -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+UseLargePages -Xmx16384M -Xms16384M -verbose:gc -Xloggc:gc.log.`date +%Y%m%d%H%M%S` -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+AggressiveOpts
Pegasystems Inc.
US
I assume you are running Oracle JDK, right? Here is a good starting point: Recommended Starting 64bit Hotspot JVM Arguments. Also send your jvm arguments for review.
Updated: 1 Feb 2016 20:00 EST
Amazon Corp
US
We did go through the same recommendations during the initial setup. Here is what we did
#HEAP
JAVA_OPTS=" -Xms16384m -Xmx16384m "
#PERM SIZE 1/8 - 1/4 HEAP
JAVA_OPTS="$JAVA_OPTS -XX:PermSize=1024m -XX:MaxPermSize=2048m "
#NEW SIZE 1/4 HEAP
JAVA_OPTS="$JAVA_OPTS -XX:NewSize=2048m -XX:MaxNewSize=2048m "
#PEGA Recommended GC Settings: https://docs-previous.pega.com/performance/jvm-options-oracle-java-hotspot-virtual-machine:
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC "
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC "
JAVA_OPTS="$JAVA_OPTS -XX:+CMSParallelRemarkEnabled "
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError "
JAVA_OPTS="$JAVA_OPTS -XX:+UseCompressedOops "
JAVA_OPTS="$JAVA_OPTS -XX:+UseCMSInitiatingOccupancyOnly "
JAVA_OPTS="$JAVA_OPTS -XX:CMSInitiatingOccupancyFraction=80 "
JAVA_OPTS="$JAVA_OPTS -XX:TargetSurvivorRatio=90 "
This has been working fine, and the heap usage is very much in control. But for some reason when I switched to using EAR instead of prweb.war, I started seeing the error messages that I pasted. Is this something to do with the compatibility?
Pegasystems Inc.
US
not sure what you mean by compatibility. I would suggest you adjust XX:CMSInitiatingOccupancyFraction to 70 and report back if this makes difference. See this Oracle link for more details: Concurrent Mark Sweep (CMS) Collector
If this was not promotion failuer, think about why the NewSize is very small 2GB (new) vs 14GB (old)? I guess it will be solved by-XX:NewSize=4096m -XX:MaxNewSize=4096m.
But if it was promotion failuer, exact Kevin said.
Amazon Corp
US
I tried both of them. But I still see the same messages.
[GC (Allocation Failure) [ParNew: 1887488K->209664K(1887488K), 0.2473887 secs] 1933992K->284439K(8178944K), 0.2476743 secs] [Times: user=0.84 sys=0.03, real=0.25 secs]
[GC (Allocation Failure) [ParNew: 1887488K->209664K(1887488K), 0.2554664 secs] 1962263K->367354K(8178944K), 0.2556495 secs] [Times: user=0.85 sys=0.06, real=0.26 secs]
[GC (Allocation Failure) [ParNew: 1887488K->164736K(1887488K), 0.1227788 secs] 2045178K->322427K(8178944K), 0.1229600 secs] [Times: user=0.47 sys=0.00, real=0.13 secs]
[GC (Allocation Failure) [ParNew: 1842560K->209664K(1887488K), 0.1226658 secs] 2000251K->367362K(8178944K), 0.1228389 secs] [Times: user=0.47 sys=0.00, real=0.12 secs]
[GC (Allocation Failure) [ParNew: 1887488K->158297K(1887488K), 0.1270683 secs] 2045186K->315996K(8178944K), 0.1272290 secs] [Times: user=0.45 sys=0.00, real=0.12 secs]
[GC (Allocation Failure) [ParNew: 1836121K->115590K(1887488K), 0.2118572 secs] 1993820K->368765K(8178944K), 0.2120373 secs] [Times: user=0.66 sys=0.06, real=0.21 secs]
[GC (Allocation Failure) [ParNew: 1793414K->34908K(1887488K), 0.1057507 secs] 2046589K->327112K(8178944K), 0.1059065 secs] [Times: user=0.33 sys=0.01, real=0.11 secs]
[GC (Allocation Failure) [ParNew: 1712732K->8865K(1887488K), 0.0388111 secs] 2004936K->302147K(8178944K), 0.0389582 secs] [Times: user=0.11 sys=0.00, real=0.04 secs]
Pegasystems Inc.
US
what version of jdk are you using? As a test, can you even lower XX:CMSInitiatingOccupancyFraction to 50 (which is very low) to see the allocation failure disappears? Also can you upload your verbose gc log?
Pegasystems Inc.
US
can u revert back the XX:CMSInitiatingOccupancyFraction to 80 and try the next two suggestions from Redhat? starting one at a time?
Accepted Solution
Pegasystems Inc.
US
G1 is supposed to replace CMS collector eventually, since you are using java 1.8, you probably want to go to that route. All our current recommendation is based on field experience with java 6/7, so it will take some time for pega to actually collect enough data for java 1.8 systems, both internally and externally. For example, Redhat would recommend the following to start with - but ultimately jvm tuning should be really based on your own applications with sufficient load tests with realistic use case scenarios.
-server -XX:+DoEscapeAnalysis -XX:+UseCompressedOops -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+UseLargePages -Xmx16384M -Xms16384M -verbose:gc -Xloggc:gc.log.`date +%Y%m%d%H%M%S` -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+AggressiveOpts
Pegasystems Inc.
US
FYI: this is from Redhat support knowledge base, several other things to try:
- To account for large variances in application object allocation rates, test the
-XX:CMSIncrementalSafetyFactor=NN(default 10) option to start the concurrent collection NN% sooner than the calculated time when -XX:+CMSIncrementalMode is used. For example: - Consider using the G1 collector instead on Java 7. This is the successor to the CMS collector and helps improve CMS's most notable issues with fragmentation.