Question
Accenture
IN
Last activity: 14 Jun 2016 5:19 EDT
Pega 7.1.7 Application is facing OutOfMemory Issue even though it is configured with 4GB RAM
Hi Guys,
My application was stable for months till yesterday, and now we are facing OutOfMemory issue. Apart form that i can see some other error as java.lang.NoClassDefFoundError: com/pega/pegarules/pub/context/AbandonProcessingException
Could you please help me on this?
Please find the below logs.
<May 31, 2016 6:08:40 AM CEST> <Critical> <Health> <BEA-310003> <Free memory in the server is 4,283,368 bytes. There is danger of OutOfMemoryError>
2016-05-31 08:48:58,203 [fault (self-tuning)'] [ STANDARD] [ ] [ ] ( internal.async.Agent) INFO - System date: Tue May 31 08:48:58 CEST 2016 Total memory: 3,113,877,504 Free memory: 45,586,144 Requestor Count: 88 Shared Pages memory usage: 0%
<May 31, 2016 8:50:42 AM CEST> <Critical> <Health> <BEA-310003> <Free memory in the server is 2,094,624 bytes. There is danger of OutOfMemoryError>
2016-05-31 08:58:58,222 [fault (self-tuning)'] [ STANDARD] [ ] [ CMBOM:01.02.01] ( internal.async.Agent) INFO - System date: Tue May 31 08:58:58 CEST 2016 Total memory: 3,113,877,504 Free memory: 32,436,784 Requestor Count: 88 Shared Pages memory usage: 0%
Hi Guys,
My application was stable for months till yesterday, and now we are facing OutOfMemory issue. Apart form that i can see some other error as java.lang.NoClassDefFoundError: com/pega/pegarules/pub/context/AbandonProcessingException
Could you please help me on this?
Please find the below logs.
<May 31, 2016 6:08:40 AM CEST> <Critical> <Health> <BEA-310003> <Free memory in the server is 4,283,368 bytes. There is danger of OutOfMemoryError>
2016-05-31 08:48:58,203 [fault (self-tuning)'] [ STANDARD] [ ] [ ] ( internal.async.Agent) INFO - System date: Tue May 31 08:48:58 CEST 2016 Total memory: 3,113,877,504 Free memory: 45,586,144 Requestor Count: 88 Shared Pages memory usage: 0%
<May 31, 2016 8:50:42 AM CEST> <Critical> <Health> <BEA-310003> <Free memory in the server is 2,094,624 bytes. There is danger of OutOfMemoryError>
2016-05-31 08:58:58,222 [fault (self-tuning)'] [ STANDARD] [ ] [ CMBOM:01.02.01] ( internal.async.Agent) INFO - System date: Tue May 31 08:58:58 CEST 2016 Total memory: 3,113,877,504 Free memory: 32,436,784 Requestor Count: 88 Shared Pages memory usage: 0%
<May 31, 2016 9:06:43 AM CEST> <Critical> <Health> <BEA-310003> <Free memory in the server is 3,540,184 bytes. There is danger of OutOfMemoryError>
2016-05-31 09:08:58,241 [fault (self-tuning)'] [ STANDARD] [ ] [ ] ( internal.async.Agent) INFO - System date: Tue May 31 09:08:58 CEST 2016 Total memory: 3,113,877,504 Free memory: 28,057,008 Requestor Count: 88 Shared Pages memory usage: 0%
2016-05-31 09:18:58,262 [fault (self-tuning)'] [ STANDARD] [ ] [ CMBOM:01.02.01] ( internal.async.Agent) INFO - System date: Tue May 31 09:18:58 CEST 2016 Total memory: 3,113,877,504 Free memory: 28,570,504 Requestor Count: 88 Shared Pages memory usage: 0%
2016-05-31 09:29:05,492 [fault (self-tuning)'] [ STANDARD] [ ] [ ] ( internal.async.Agent) INFO - System date: Tue May 31 09:29:05 CEST 2016 Total memory: 3,113,877,504 Free memory: 83,615,048 Requestor Count: 90 Shared Pages memory usage: 0%
2016-05-31 09:34:41,412 [fault (self-tuning)'] [ ] [ ] [ ] (ngineinterface.service.HttpAPI) ERROR xxxxxxxxxxxxxxx - Unexpected error during processing
java.lang.NoClassDefFoundError: com/pega/pegarules/pub/context/AbandonProcessingException
at com.pega.pegarules.session.internal.engineinterface.service.HttpAPI.postLockCleanup(HttpAPI.java:4166)
at com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest(EngineAPI.java:338)
at com.pega.pegarules.session.internal.engineinterface.service.HttpAPI.invoke(HttpAPI.java:839)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:315)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)
at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:225)
at sun.reflect.GeneratedMethodAccessor133.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:216)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:238)
at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at com.oracle.pitchfork.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:34)
at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
at com.oracle.pitchfork.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:42)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy68.invokeEngine(Unknown Source)
at com.pega.pegarules.internal.etier.ejb.EngineBMT_h449u3_ELOImpl.__WL_invoke(Unknown Source)
at weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:39)
at com.pega.pegarules.internal.etier.ejb.EngineBMT_h449u3_ELOImpl.invokeEngine(Unknown Source)
at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngineInner(JNDIEnvironment.java:278)
at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngine(JNDIEnvironment.java:223)
at com.pega.pegarules.web.impl.WebStandardImpl.makeEtierRequest(WebStandardImpl.java:485)
at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:290)
at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:216)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:265)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:118)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.jav a:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3750)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Pegasystems Inc.
AU
Hello Chennareddy,
Could you please share the complete PegaRULES log file? it will help us to understand the pattern for the OOM behaviour. I believe this requires an investigation, hence would suggest you to create a SR with complete PegaRULES, Alert and Application server logs and in addition heap dumps would help for the analysis.
Pegasystems Inc.
GB
In addition to the PRPC Logs/Alerts: The Garbage-Collection ("GC") logs , if available will be helpful here; and we also need the JVM startup options - to make sure the Heap Size is correctly set to 4Gb.
One other thing which will be useful: are you able to trigger a Heap Dump when the OOM errors occur ? (Warning: the heap dump will be at least as big your heap size; and will slow down your system whilst it is being generated).
Pegasystems Inc.
US
Hi Chenna,
When you are in a reply, click on Use advanced editor in the top right of the reply box.
This will refresh your screen and you will see the option to Attach on the bottom right of the reply box.
Hope that helps!
Pegasystems Inc.
IN
Request you to check if you are able to attach the log files to your post. Do let me know if you still face issues.
~Vidyaranjan A V | Community Moderator | Pegasystems Inc.
Accenture
IN
Hi Vidyaranjan Av,
No i don't see any option still.
Updated: 2 Jun 2016 6:41 EDT
Pegasystems Inc.
GB
EDIT: I see you are just missing the option for 'Use Advanced Editor' altogether in fact ? Can you try editing your original post (or create a new test post) - do you get the option for that ?
Hi Chenna,
With regard to not being able to attach: I just tried this on this post - and I was given the option after switching to the 'advanced editor':
Here's my screenshots:
Do you see extra items on the toolbar after switching to 'advanced editor' or not ?
You could also try editing your *original* post - rather than commenting - I find sometimes the COMMENT box doesn't always provide the same options as a 'new/edit' post ?
Cheers
John
Pegasystems Inc.
IN
It looks like raising an SR as suggested by John Pritchard-williams would be the best way to get assistance with your issue. Once you go ahead and raise an SR, I request you to share the SR number here, so that we can track it for you within the thread.
~Vidyaranjan A V | Community Moderator | Pegasystems Inc.
Pegasystems Inc.
IN
Did you happen to raise an SR for this issue? If yes, share the details of the SR number here, so that we can track it for you within the thread.
~Vidyaranjan A V | Community Moderator | Pegasystems Inc.
Pegasystems Inc.
IN
Thank you for sharing the SR details, I have tagged it along with the description of the original question. This will help in tracking the SR better.
~Vidyaranjan A V | Community Moderator | Pegasystems Inc.
Updated: 2 Jun 2016 6:36 EDT
Pegasystems Inc.
GB
Hi Chenna,
I would not advise you actually attach your logfiles here : you should open an SR and provide them that way: they'll need to be looked at in detail.
(Also: if you *do* decide to attach them here - you need to be sure you 'scrub' them of any site-specific information etc).
Thanks !
John
Pegasystems Inc.
IN
Thank you for the valuable inputs in assisting Chenna Reddy with the file attach option. I am working with him on the same, he should be able to attach the files very soon.
Regards,
Vidyaranjan A V | Community Moderator | Pegasystems Inc.
Hello,
I had a look into the associated SR and realised that it has been closed due to lack of inputs (files were deleted without backup during server down). I will be closing this discussion too. Please reopen it at any time in the future should you wish to.
Regards,
Lochan | Online Community Moderator | Pegasystems Inc.