Question
TATA CONSULTANCY SERVICES
IN
Last activity: 27 Jan 2016 23:47 EST
Hung Threads - Does Pega uses SoapConnectorThreadPool ?
Hi All,
We are observing few hung threads in the JVM & found that those hung threads are related to Pega. As per the Stacktrace looks like - Pega OOTB MBean operation to get the RequestorList Size was invoked.
Few Questions........
1) Does Pega make use of Websphere - SoapConnectorThreadPool ?? If yes, in which all Scenarios/functionalities does Pega makes use of this Thread Pool. I'm only aware of WebContainer Threads.
2) Is there any recommendation from pega on what size does the SoapConnectorThreadPool should be ?
3) Looking at this log - can anyone tell whether thsi MBean Operation invoked manually (I mean may be someone was using SMA at that timeframe ?) or what other actions/applications would have triggered this Request ?
at com.pega.pegarules.management.internal.RequestorManagement.RequestorListWithSize(RequestorManagement.java:133)
[12/22/15 21:37:23:001 MST] 00000003 ThreadMonitor W CWWSR0605W: Thread "SoapConnectorThreadPool : 23" (00000072) has been active for 603481 milliseconds a
nd may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.estimateDataSize(ClipboardPageImpl.java:5021)
Hi All,
We are observing few hung threads in the JVM & found that those hung threads are related to Pega. As per the Stacktrace looks like - Pega OOTB MBean operation to get the RequestorList Size was invoked.
Few Questions........
1) Does Pega make use of Websphere - SoapConnectorThreadPool ?? If yes, in which all Scenarios/functionalities does Pega makes use of this Thread Pool. I'm only aware of WebContainer Threads.
2) Is there any recommendation from pega on what size does the SoapConnectorThreadPool should be ?
3) Looking at this log - can anyone tell whether thsi MBean Operation invoked manually (I mean may be someone was using SMA at that timeframe ?) or what other actions/applications would have triggered this Request ?
at com.pega.pegarules.management.internal.RequestorManagement.RequestorListWithSize(RequestorManagement.java:133)
[12/22/15 21:37:23:001 MST] 00000003 ThreadMonitor W CWWSR0605W: Thread "SoapConnectorThreadPool : 23" (00000072) has been active for 603481 milliseconds a
nd may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.estimateDataSize(ClipboardPageImpl.java:5021)
at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.estimateDataSize(ClipboardPropertyImpl.java:862)
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.estimateDataSize(ClipboardPageImpl.java:4993)
at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.estimateDataSize(ClipboardPropertyImpl.java:862)
at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.estimateDataSize(ClipboardPropertyImpl.java:855)
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.estimateDataSize(ClipboardPageImpl.java:4993)
at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.estimateDataSize(ClipboardPropertyImpl.java:862)
at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.estimateDataSize(ClipboardPropertyImpl.java:855)
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.estimateDataSize(ClipboardPageImpl.java:4993)
at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.estimateDataSize(ClipboardPropertyImpl.java:862)
at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.estimateDataSize(ClipboardPropertyImpl.java:855)
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.estimateDataSize(ClipboardPageImpl.java:4993)
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.estimateDataSize(ClipboardPageImpl.java:4872)
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.estimateDataSize(ClipboardPageImpl.java:4868)
at com.pega.pegarules.session.internal.mgmt.base.handler.UserPageHandler.estimateDataSize(UserPageHandler.java:201)
at com.pega.pegarules.session.internal.mgmt.base.AbstractPageDirectory.estimatePageDirectoryDataSize(AbstractPageDirectory.java:1345)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.estimateDataSize(ThreadPageDir.java:808)
at com.pega.pegarules.session.internal.mgmt.PRThreadImpl.estimateDataSize(PRThreadImpl.java:1112)
at com.pega.pegarules.session.internal.mgmt.base.RequestorPageDir.estimateDataSize(RequestorPageDir.java:214)
at com.pega.pegarules.session.internal.mgmt.PRRequestorImpl.estimateDataSize(PRRequestorImpl.java:1559)
at com.pega.pegarules.management.api.PegaRequestorManagementImpl$2.run(PegaRequestorManagementImpl.java:281)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doUnsafeAccessToUnlockedRequestor(PRSessionProviderImpl.java:509)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doUnsafeAccessToUnlockedRequestor(PRSessionProviderImpl.java:465)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doUnsafeAccessToUnlockedRequestor(PRSessionProviderImpl.java:2006)
at com.pega.pegarules.management.api.PegaRequestorManagementImpl.getRequestorList(PegaRequestorManagementImpl.java:218)
at com.pega.pegarules.management.internal.RequestorManagementAPI.RequestorList(RequestorManagementAPI.java:308)
at com.pega.pegarules.management.internal.RequestorManagementAPI.RequestorList(RequestorManagementAPI.java:284)
at com.pega.pegarules.management.internal.RequestorManagementAPI.RequestorListWithSize(RequestorManagementAPI.java:276)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.pega.pegarules.management.internal.MB2EngineAccess.invoke(MB2EngineAccess.java:169)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invoke_privact(EngineImpl.java:364)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.run(EngineImpl.java:349)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1177)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:926)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:777)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:323)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)
at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:225)
at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:216)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:238)
at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)
at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.invokeEngine(Unknown Source)
at com.pega.pegarules.management.internal.MBeanHelper.invokeOperation(MBeanHelper.java:77)
at com.pega.pegarules.management.internal.RequestorManagement.RequestorListWithSize(RequestorManagement.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:69)
at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:272)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1152)
at java.security.AccessController.doPrivileged(AccessController.java:384)
at com.ibm.oti.security.CheckedAccessControlContext.securityCheck(CheckedAccessControlContext.java:30)
at sun.misc.JavaSecurityAccessWrapper.doIntersectionPrivilege(JavaSecurityAccessWrapper.java:41)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1146)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:999)
at com.pega.pegarules.priv.management.PegaModelMBean.invoke(PegaModelMBean.java:121)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:847)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:783)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1346)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1239)
at com.ibm.ws.management.connector.AdminServiceDelegator.invoke(AdminServiceDelegator.java:181)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.ibm.ws.management.connector.soap.SOAPConnector.invoke(SOAPConnector.java:464)
at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java:297)
at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java:65)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:733)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:522)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)
Thanks
Hari Kumar Alampuru
Pegasystems Inc.
IN
Hi Hari,
Can you please share following logs during the time when you see hung threads:
-PegaRULES
-PegaALERT
-Thread Dump
You can play around following IBM parameter :
com.ibm.websphere.management.soapThreads
This controls the SoapConnectorThreadPool
TATA CONSULTANCY SERVICES
IN
Due to Large File Sizes, will not be able to share the requested logs.
It would be great if anyone can help me in answering the first 2 questions which I mentioned in my initial thread.
1) Does Pega make use of Websphere - SoapConnectorThreadPool ?? If yes, in which all Scenarios/functionalities does Pega makes use of this Thread Pool. I'm only aware of WebContainer Threads. \
2) Is there any recommendation from pega on what size does the SoapConnectorThreadPool should be ?
Thanks
Hari Kumar Alampuru
Hi Hari, It looks like a SMA operation(mbean call from SMA) that took a long time to complete. How is your SMA configured for this node? Is SMA using remote Connection to connect to Pega? this remote connection uses SOAP connector port. above is just a monitoring message and the operation would have completed if given enough time. are you consistently able to reproduce this problem?
TATA CONSULTANCY SERVICES
IN
Hi Gopi,
SMA is using remote connection to connect to Pega. Yes, that was the only an instance where we observed this hung thread but didn't came across it after that.
After our analysis, we found that Hung Thread was possibly due to HIGH GC running on the Pega Servers.
Just for my understanding can you/anyone please confirm if "SoapConnectorThreadPool" threads are created even for Connector Rules that are invoked from Pega instance ? and Is there any recommendation from pega on what size does the SoapConnectorThreadPool should be ?
Thanks
Hari Kumar Alampuru
Pegasystems Inc.
US
no SoapConnectorThreadPool is not used for connector rules. Like Gopi said, it was JMX calls from SMA that uses this pool. In general, SMA to PRPC is very low volume, you should have no need to adjust the SoapConnectorThreadPool size. I think your issue is most likely due to the performance issue you mentioned for that particular PRPC instance such that the call from SMA to that instance never returns. However, if you really want to adjust, here is link I found from IBM developer community: How can I set the maximum capacity for the SoapConnectorThreadPool? - dWAnswers