Thread dumps due to java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
Pega 7.3.1, we are getting frequent thread dumps due to ReentrantReadWriteLock.
Do any one have any idea? and how to get rid of these
Pega 7.3.1, we are getting frequent thread dumps due to ReentrantReadWriteLock.
Do any one have any idea? and how to get rid of these
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@5b17fe6e (running in native)
owned by TP_http : 368 Id=2637
BlockedCount : 3, BlockedTime : -1, WaitedCount : 9, WaitedTime : -1
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:945)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1258)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1126)
at com.pega.pegarules.session.internal.mgmt.base.RequestorThreadSync.lockAttempt(RequestorThreadSync.java:460)
at com.pega.pegarules.session.internal.mgmt.PRRequestorImpl.lockAttempt(PRRequestorImpl.java:1009)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.getLockOnRequestor(PRSessionProviderImpl.java:1435)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1061)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:930)
at com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest(EngineAPI.java:340)
at com.pega.pegarules.session.internal.engineinterface.service.HttpAPI.invoke(HttpAPI.java:863)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:331)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:274)
at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:225)
at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:507)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:246)
at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)
at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.invokeEngine(Unknown Source)
at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngineInner(JNDIEnvironment.java:278)
at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngine(JNDIEnvironment.java:223)
at com.pega.pegarules.web.impl.WebStandardImpl.makeEtierRequest(WebStandardImpl.java:678)
at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:394)
at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:507)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:273)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:121)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:92)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1232)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:781)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:480)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:940)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1070)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.readyInboundPostHandshake(SSLConnectionLink.java:718)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyHandshakeCompletedCallback.complete(SSLConnectionLink.java:414)
at com.ibm.ws.ssl.channel.impl.SSLUtils.handleHandshake(SSLUtils.java:1074)
at com.ibm.ws.ssl.channel.impl.SSLHandshakeIOCallback.complete(SSLHandshakeIOCallback.java:87)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:816)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
***Moderator Edit-Vidyaranjan: Updated SR details***
Accepted Solution
Pegasystems Inc.
US
Upon reviewing the associated Support Request, the following was the explanation for the reported behavior:
The pyOnBeforeWindowClose activty is for cleaning up requestors when a user closes the browser window instead of logging out. When the user closed the browser the session state is trying to be updated within hazelcast but the hazelcast threads is timing out. This can happen if there are nodes frequently joining and leaving the cluster. These timeouts are not causing any functional impact. There is another daemon running in the background to mark these closed sessions as disconnected.
Recommendation is to ignore these OperationTimeoutExceptions that sporadically occur from pyOnbeforeWindowClose activity
Pegasystems Inc.
IN
Can you please share PegaRULES when you observed this issue/log message?
Ford Motor Company
IN
Sorry Arvind, I may not be able to share the complete log as it will have proprietary information.
But in another thread with similar description we see the Pega activities in the below trace..
com.pegarules.generated.activity.ra_action_pzsetrequestorstateasdisconnected_bb1ea6c5c00e81502f172683d9416652.perform(ra_action_pzsetrequestorstateasdisconnected_bb1ea6c5c00e81502f172683d9416652.java:70)
at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3597)
at com.pega.pegarules.session.internal.mgmt.Executable.invokeActivity(Executable.java:10845)
at com.pegarules.generated.activity.ra_action_pyonbeforewindowclose_6e4556eb02e0b2f29ba0650d52c31657.step2_circum0
Pegasystems Inc.
IN
You can raise SR so that you can share logs to specific Support personnel and he/she assists you.
You can verify running thread which never completed to see what that thread is doing.
Ford Motor Company
IN
Thanks.
Meanwhile I commented the 2nd step of the activity pyonbeforewindowclose, and it has reduced the thread dumps. and temporarily it helped in performance.
Will wait for permanent fix via SR.
Accepted Solution
Pegasystems Inc.
US
Upon reviewing the associated Support Request, the following was the explanation for the reported behavior:
The pyOnBeforeWindowClose activty is for cleaning up requestors when a user closes the browser window instead of logging out. When the user closed the browser the session state is trying to be updated within hazelcast but the hazelcast threads is timing out. This can happen if there are nodes frequently joining and leaving the cluster. These timeouts are not causing any functional impact. There is another daemon running in the background to mark these closed sessions as disconnected.
Recommendation is to ignore these OperationTimeoutExceptions that sporadically occur from pyOnbeforeWindowClose activity
American International Group Inc
US
Experiencing same issue Pega 8.2 env. Could you please let us know the final resolution of SR-C58942?
"WebContainer : 7" Id=1324 in TIMED_WAITING on lock=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2d8ce690 (running in native)
owned by WebContainer : 5 Id=1211
BlockedCount : 13, BlockedTime : -1, WaitedCount : 34984, WaitedTime : -1
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:945)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1258)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1126)
at com.pega.pegarules.session.internal.mgmt.base.RequestorThreadSync.lockAttempt(RequestorThreadSync.java:463)
at com.pega.pegarules.session.internal.mgmt.PRRequestorImpl.lockAttempt(PRRequestorImpl.java:1025)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.getLockOnRequestor(PRSessionProviderImpl.java:1479)