Spike in WAS threads resulting in node unavailability

Question

RanjithkumarKasula

Member since 2016

2 posts

Posted: Feb 6, 2016

Last activity: Feb 9, 2016

Posted: 6 Feb 2016 20:27 EST
Last activity: 9 Feb 2016 21:05 EST

Closed

Solved

Spike in WAS threads resulting in node unavailability

Report

Issue Description:

We have been having an issue with the JVM web container threads maxing out all of a sudden where in the threads spike up to a max of 100 from around 6 threads in couple of seconds.

Observations:

The JVM’s spike up instantly and runs on max threads for about 4-5 minutes and recovers on their own.
Only few JVM’s are getting impacted (mostly 4 of 24 in total).
There were no long running DB transactions, thread blocks; CPU utilization is well within limit(less than 60%) at the database side.
This issue is not occurring consistently on the same JVM’s.
This is occurring intermittently, not every day.

Below are the observations from our PEGA logs:

ParkNanos

Thread[WebContainer : 27,5,main]

sun.misc.Unsafe.park(Native Method)

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:222)

Issue Description:

We have been having an issue with the JVM web container threads maxing out all of a sudden where in the threads spike up to a max of 100 from around 6 threads in couple of seconds.

Observations:

The JVM’s spike up instantly and runs on max threads for about 4-5 minutes and recovers on their own.
Only few JVM’s are getting impacted (mostly 4 of 24 in total).
There were no long running DB transactions, thread blocks; CPU utilization is well within limit(less than 60%) at the database side.
This issue is not occurring consistently on the same JVM’s.
This is occurring intermittently, not every day.

Below are the observations from our PEGA logs:

ParkNanos

Thread[WebContainer : 27,5,main]

sun.misc.Unsafe.park(Native Method)

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:222)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:916)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1235)

at com.pega.jsr166backport.java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:422)

at com.pega.pegarules.session.internal.mgmt.base.RequestorThreadSync.lockAttempt(RequestorThreadSync.java:403)

at com.pega.pegarules.session.internal.mgmt.PRRequestorImpl.lockAttempt(PRRequestorImpl.java:842)

at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:930)

at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:792)

at com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest(EngineAPI.java:328)

at com.pega.pegarules.session.internal.engineinterface.service.HttpAPI.invoke(HttpAPI.java:703)

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:312)

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)

at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:221)

at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

at java.lang.reflect.Method.invoke(Method.java:611)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:349)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:390)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingException(PRBootstrap.java:412)

at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)

at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.invokeEngine(Unknown Source)

at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngineInner(JNDIEnvironment.java:277)

at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngine(JNDIEnvironment.java:222)

at com.pega.pegarules.web.impl.WebStandardImpl.makeEtierRequest(WebStandardImpl.java:373)

at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:266)

at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

at java.lang.reflect.Method.invoke(Method.java:611)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:349)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:390)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:439)

at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:99)

at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:89)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1694)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:970)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:508)

at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)

at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)

at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)

at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)

at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454)

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:516)

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:307)

at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)

at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)

at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)

at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)

at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)

at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)

at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:816)

at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)

at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)

sun.misc.Unsafe.park(Native Method)

Thread[WebContainer : 23,5,main]