REST service times out SocketTimeoutException (PRPC 6.3sp1)
REST service times out with an error as below :-
- Mon Oct 12 13:25:00 IST 2015:INFO:Error getting response for [.:]; java.net.SocketTimeoutException: Read timed out
On analyzing the SMA, I am seeing the Idle count of threads as "0" and only then this issue comes up.. This is a stateful authenticated service. Below screenshot from SMA.
I had purposefully set the max idle count as "0" and Max Active count as "0" in the pooling tab. This should not affect as this is a stateful-authneticated package. I also made it as - setting max active to 1 and the issue comes up again.
Any idea on how to debug this ?
I checked the weblogic logs and saw the below error :-
####<Oct 9, 2015 12:14:03 AM PDT> <Error> <WebLogicServer> <stage2cpp087> <stage2cpp087_unify_L2_1> <[ACTIVE] ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1444374843831> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "608" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 608245 ms
REST service times out with an error as below :-
- Mon Oct 12 13:25:00 IST 2015:INFO:Error getting response for [.:]; java.net.SocketTimeoutException: Read timed out
On analyzing the SMA, I am seeing the Idle count of threads as "0" and only then this issue comes up.. This is a stateful authenticated service. Below screenshot from SMA.
I had purposefully set the max idle count as "0" and Max Active count as "0" in the pooling tab. This should not affect as this is a stateful-authneticated package. I also made it as - setting max active to 1 and the issue comes up again.
Any idea on how to debug this ?
I checked the weblogic logs and saw the below error :-
####<Oct 9, 2015 12:14:03 AM PDT> <Error> <WebLogicServer> <stage2cpp087> <stage2cpp087_unify_L2_1> <[ACTIVE] ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1444374843831> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "608" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 608245 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
com.pega.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:743)
com.pega.pegarules.session.internal.mgmt.RequestorPool.borrowRequestor(RequestorPool.java:455)
com.pega.pegarules.session.internal.mgmt.RequestorPoolManager.borrowRequestor(RequestorPoolManager.java:61)
com.pega.pegarules.session.internal.PRSessionProviderImpl.borrowRequestor(PRSessionProviderImpl.java:573)
com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:883)
com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:771)
com.pega.pegarules.integration.engine.internal.services.ServiceAPI.getServiceMethod(ServiceAPI.java:2624)
com.pega.pegarules.integration.engine.internal.services.http.HTTPService.invoke(HTTPService.java:199)
com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:312)
com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)
com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:221)
sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:349)
com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:390)
com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingException(PRBootstrap.java:412)
com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)
com.pega.pegarules.internal.etier.ejb.EngineBMT_h449u3_ELOImpl.__WL_invoke(Unknown Source)
weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:33)
com.pega.pegarules.internal.etier.ejb.EngineBMT_h449u3_ELOImpl.invokeEngine(Unknown Source)
com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngineInner(JNDIEnvironment.java:277)
com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngine(JNDIEnvironment.java:222)
com.pega.pegarules.web.impl.WebStandardImpl.makeEtierRequest(WebStandardImpl.java:400)
com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:278)
sun.reflect.GeneratedMethodAccessor201.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:349)
com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:390)
com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:439)
com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:118)
com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:89)
javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:243)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3432)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3402)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2285)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2201)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1572)
weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
>
***Updated by Moderator: Marissa to add Doc Bug group tag***
**Moderation Team has archived post**
This post has been archived for educational purposes. Contents and links will no longer be updated. If you have the same/similar question, please write a new post.
Accepted Solution
PayPal
IN
Response via SR
----** posting here for the benefit of the community **** --
As discussed the stateless authenticated requestors are also pooled , though the documentation seems incorrect. (
Doc BUG-223565 raised.
)
As confirmed by product specialist in code it looks like for stateless authenticated requestors we also rely on requestor pooling, the only difference is while returning a requestor, we return a bogus requestor-id(a blank string) back to the pool , and actually destroy the requestor, we return a blank id to maintain the SMA counters in sync.
In your case you’re giving MaxActive 0 ,MaxIdle 0 and MAxWait 0, this configuration itself is incorrect, as the code takes it literally to be zero, requestors will be available in the pool ever , hence the application requests for a new requestor for a pool and waits till the requestor/Containers Work Manager times out. In case if maxactive is negative it will allow unlimited requestors.
Response via SR
----** posting here for the benefit of the community **** --
As discussed the stateless authenticated requestors are also pooled , though the documentation seems incorrect. (
Doc BUG-223565 raised.
)
As confirmed by product specialist in code it looks like for stateless authenticated requestors we also rely on requestor pooling, the only difference is while returning a requestor, we return a bogus requestor-id(a blank string) back to the pool , and actually destroy the requestor, we return a blank id to maintain the SMA counters in sync.
In your case you’re giving MaxActive 0 ,MaxIdle 0 and MAxWait 0, this configuration itself is incorrect, as the code takes it literally to be zero, requestors will be available in the pool ever , hence the application requests for a new requestor for a pool and waits till the requestor/Containers Work Manager times out. In case if maxactive is negative it will allow unlimited requestors.
This is the desired behavior and the documentation is incorrect if it says stateless/authenticated requestors do not depend on the pool. This is the same behavior for stateful services as well.
Please note that there are 2 phases in execution of a soap service
-In the first phase we open up the service package - for this we use requestor pooling
-In the second phase we execute the service activity and we dont use requestor pooling here
So overall in stateful case we use requestor pooling
Updated: 12 Oct 2015 4:48 EDT
Pegasystems Inc.
GB
Hi Nishan,
So as far as I can tell here: the situation you are getting here (a long-running thread waiting to 'borrow' a requestor from the pool):
java.lang.Object.wait(Object.java:503)
com.pega.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:743)
com.pega.pegarules.session.internal.mgmt.RequestorPool.borrowRequestor(RequestorPool.java:455)
[...]
Is consistent with the fact that you have sized the pool at zero - there are no requestors in the pool to borrow....
Basically I think we need to verify this statement:
//
[The sizing of the Requestor Pool] should not affect as this is a stateful-authneticated package
//
Because if this is the case (I'm not saying it is , or is not at this point) - then wouldn't that imply that 'stateful requestors' cannot be pooled at all ?
What is the behaviour you are expecting here ? That 'stateful requestors' are not pooled at all ? Or they that they are pooled, and their state gets perisisted alongside the requestor when it is returned to the pool ?
I have to be honest here: I have only every worked with 'stateless' requestors with regard to pooling - so I have never thought about what the behaviour of stateful requestors should be with regard to pooling.....
Updated: 12 Oct 2015 5:39 EDT
PayPal
IN
Hi John,
As per help, this is what stateful-authenticated with regard to pooling states ( Pooling tab is disabled for stateful / authenticated ):-
Use the Pooling tab to configure a requestor pool for the services in this service package when they are stateless and unauthenticated. This tab is disabled if, on the Context tab, the Processing Mode field is set to Stateful or the Requires authentication? option is selected.
If this is the case, why would the pool count number come into picture ? ( I see this contradiction in the saved XML into DB as well .. there again for stateful-auth , I am seeing the value being read from pooling tab as opposed to being null since this is a stateful-auth package).
Regards,
Nishan
Pegasystems Inc.
US
Just ran across this thread, did you open an SR for this? If so, could you please let us know the number by replying and we can track this for you.
Thanks!
Pegasystems Inc.
US
Sounds great Nishan Ansari! Definitely keep us updated!
Just to add some points related to topic being discussed, Stateful+Authenticated service execution includes the following steps :
1) will borrow a requestor from pool (unauthenticated one)
2) tries to authenticate the requestor
3) maps inbound data, invokes service activity& maps outbound data
4) notifies the pool that it will not return the requestor to pool, so that pool can perform house keeping job
5) requestor idle timeout defines the lifetime of that particular requestor (which will be used for subsequent service requests processing)
Updated: 27 Oct 2015 5:53 EDT
PayPal
IN
Hi Chandra,
If that is the case, if I set the pool idle count to 1 and if that 1 requester is borrowed for processing; if the pool does not do the house keeping job of re generating the borrowed requestor, this error MAY come up right ?
Pegasystems Inc.
US
Thanks Nishan Ansari! I've updated this thread with the SR!
PayPal
IN
Thanks Marissa Rogers
Accepted Solution
PayPal
IN
Response via SR
----** posting here for the benefit of the community **** --
As discussed the stateless authenticated requestors are also pooled , though the documentation seems incorrect. (
Doc BUG-223565 raised.
)
As confirmed by product specialist in code it looks like for stateless authenticated requestors we also rely on requestor pooling, the only difference is while returning a requestor, we return a bogus requestor-id(a blank string) back to the pool , and actually destroy the requestor, we return a blank id to maintain the SMA counters in sync.
In your case you’re giving MaxActive 0 ,MaxIdle 0 and MAxWait 0, this configuration itself is incorrect, as the code takes it literally to be zero, requestors will be available in the pool ever , hence the application requests for a new requestor for a pool and waits till the requestor/Containers Work Manager times out. In case if maxactive is negative it will allow unlimited requestors.
Response via SR
----** posting here for the benefit of the community **** --
As discussed the stateless authenticated requestors are also pooled , though the documentation seems incorrect. (
Doc BUG-223565 raised.
)
As confirmed by product specialist in code it looks like for stateless authenticated requestors we also rely on requestor pooling, the only difference is while returning a requestor, we return a bogus requestor-id(a blank string) back to the pool , and actually destroy the requestor, we return a blank id to maintain the SMA counters in sync.
In your case you’re giving MaxActive 0 ,MaxIdle 0 and MAxWait 0, this configuration itself is incorrect, as the code takes it literally to be zero, requestors will be available in the pool ever , hence the application requests for a new requestor for a pool and waits till the requestor/Containers Work Manager times out. In case if maxactive is negative it will allow unlimited requestors.
This is the desired behavior and the documentation is incorrect if it says stateless/authenticated requestors do not depend on the pool. This is the same behavior for stateful services as well.
Please note that there are 2 phases in execution of a soap service
-In the first phase we open up the service package - for this we use requestor pooling
-In the second phase we execute the service activity and we dont use requestor pooling here
So overall in stateful case we use requestor pooling