[PEGA 7.1.6] Connect-SOAP Response Timeout
Hello,
We have set our Connect-SOAP Response Timeout to 120 seconds.
In Production we had an Long-Running Connect SOAP Alert for the same Connect-SOAP rule. It was 231 seconds.
We investigated the Connect-SOAP in DEV, and had the following two behaviors:
1.) If endpoint was available, but took over 120 seconds to response, the Response Timeout properly triggered.
2.) If the endpoint was unavailable, we received an immediate response, and the resources were properly released.
What could have made our Production scenario different than the DEV scenario? Why did it hang in Production for such a long time?
I would greatly appreciate any insight you can provide on this matter.
Thank you!
Fran
Accepted Solution
Capgemini
IN
My guess is connect rule from pega is not behaving differently in dev or prod. The service provider is behaving differently. They are taking good amount of time to return endpoint failure in prod. And then pega is trying multiple time to connect to the same endpoint resulting the timeout.
-Saikat
Virtusa IT Consulting
AE
Pega-alert log of production env should help you understand the exact root cause for the problem.
Did you get a chance to look at the pega-alert logs? can you share it here?
You should have received Pega0011 alert for the reported behaviour.
Pegasystems Inc.
JP
Hello,
Thank you for your response!
See the Alert details below.
I appreciate your feedback!
- Fran
Hello,
Thank you for your response!
See the Alert details below.
I appreciate your feedback!
- Fran
2017-05-24 09:12:12,808 JST JST -- 2017-05-24 00:12:12,807 GMT*7*PEGA0020*231156*1000*ddc7a751122dcf088fc91d7a09e4052f*NA*NA*H01DE62392B8D5C81A2132CFC85C2EE45*SBA11094* 37e2faa90808b463be422f674dac58b*N*8*1304*WebContainer : 24*TABTHREAD0*com.pega.pegarules.session.internal.mgmt.Executable* Activity=ReloadSection*Rule-Obj-Activity:ReloadSection*RULE-CONNECT-SOAP INVOKEAXIS2 #20140724T100426.557 GMT Step: 15 Circum: 0**pxTotalReqCPU=0.07;pxOtherCount=5;pxRuleCount=1;pxRulesUsed=10;pxRunStreamCount=1;pxTotalReqTime=231.16;pxActivityCount=2;pxConnectElapsed=231.16;pxAlertCount=1;pxRuleFromCacheCount=1;pxRulesExecuted=3;pxOtherFromCacheCount=5;*Rule-Connect-SOAP*NA*Rule-Connect-SOAP*NA*doActivity Rule-Obj-Activity:Invoke;Connect-SOAP;RULE-OBJ-ACTIVITY DATA-PORTAL FETCHAUTOMATICNOTICES #20141210T194232.329 GMT Step: 4 Circum: 0;doActivity Rule-Obj-Activity:FetchAutomaticNotices;getStream Rule-HTML-Section:LoadAutomaticNotices;6 additional frames in stack;*pyRequestHeaders={};pyStatusMsg=com.pega.pegarules.pub.services.ResourceUnavailableException: SOAP service failed;pyStatusVal=Fail:InternalReason;pyTempPlaceHolder=TempPlaceHolder;pyServiceClient=com.pega.apache.axis2.client.ServiceClient@1403f;Operation=;Parameter=;pyObjName=pyTempDataPage;EndPointURL=;pyRequestParams=[<ns1:ActivityNoteRq xmlns:ns1EQUALS"http://www.ACORD.org/standards/PC_Surety/ACORD1/xml/"> <ns1:RqUID></ns1:RqUID> <ns1:TransactionRequestDt></ns1:TransactionRequestDt> <ns2:CurCd xsi:typeEQUALS"ns1:OpenEnum" xmlns:xsiEQUALS"http://www.w3.org/2001/XMLSchema-instance" +;pyAxisPackage=;*RULE-CONNECT-SOAP #20141209T142149.533 GMT has exceeded the threshold of 1,000 ms: 231,156 ms;pxConnectOutMapReqElapsed=1;pxConnectClientResponseElapsed=231073;pxConnectInMapReqElapsed=0;*
***reply edited by moderator:Lochan to remove proprietary information***
Pegasystems Inc.
US
Hi Fran,
Do you see any PEGA0011 alert?
| PEGA0011 | Service Total Time | Services — One of five time thresholds was exceeded during a service execution. See Testing Services and Connectors on the Integration page of the PDN. |
| PEGA0020 | Connect Total Time | A connector call to an external system has exceeded an elapsed time threshold. See How to detect lengthy connector executions and Testing Services and Connectors, located on the Integration page of the PDN. |
If you are getting Pega0020 that means the call is being made from PRPC and it took more time than expected to receive the response.
Did you try tracing the call from outside of PRPC, that is did you try hitting the endpoint from the production box using soapui or any other tool?
Thanks,
Mukul
Pegasystems Inc.
JP
Hello,
We ran a test from Pega in DEV with the same Connector. It timed out, as configured, after 120 seconds, if the response took too long. If the endpoint was unavailable, it immediately returned a response. We have not been able to replicate the scenario in PROD where it did not time out after 120 seconds, but stayed alive until the service returned a response after 231 seconds.
I would expect that, if we ran it from SOAPUI, we would have to wait 231 seconds, as there is no configured timeout in SOAPUI. However, that doesn't really help us understand why Pega's timeout did not trigger. Will it help? Can we set a timeout in SOAPUI? I think the test needs to be done in Pega to determine why Pega's timeout did not trigger. Let me know if I am misunderstanding.
If you have any further guidance, it would be greatly appreciated. We are still not able to make progress on this.
Thank you!
- Fran
Pegasystems Inc.
US
Hi Fran,
Did you compare the list of hotfixes that development has and missing in case of production env.
Second, do you see any error getting thrown after 120 seconds? May be the connection timeout error is getting masked by some other error.
Thanks,
Mukul
Pegasystems Inc.
JP
Hi Mukul,
I compared the Hot Fix list and saw no difference in Hot Fixes related to integration.
There are no other errors triggered near the 120 second timeout threshold.
Let me know if you have any other feedback.
Thank you!
Fran
Pegasystems Inc.
US
Hi Fran,
If possible, try pointing your lower (PRPC) env to the production end point URL and see if you can reproduce this issue and let us know your observations.
Thanks,
Mukul
Pegasystems Inc.
JP
Hello,
This won't be possible, as I am sure you would expect, we cannot point a DEV environment at a PROD interface.
Let me know if you have any other suggestions, as we are still unable to make progress with this issue.
Thank you!
Fran
JPMorgan Chase & Company
US
Hi,
Could you please attach the connect-soap rule screenshots ? Have you observed the same behaviour in pre-prod environment also.
Pegasystems Inc.
JP
Hello,
I have already been reprimanded once in this discussion for providing information private to the client. Therefore, I don't feel comfortable providing screenshots of client rules on a public forum.
Please let me know the specific configurations you want to check, and I will provide those.
This issue only occurred in PROD, nowhere else.
Please help with the resolution of this issue.
- Fran
Capgemini
IN
Please refer the below url. This should help you how timeout works in pega.
https://collaborate.pega.com/question/pega-721connect-soap-timeout-not-working
-Saikat
Pegasystems Inc.
JP
Hello,
Thank you for the reference.
I did not specify in this thread, but this was actually the article I came across during my initial investigation. Unfortunately, it does not shed any light on why we have the behaviour that we do in PROD.
If you have any other information, that would be appreciated.
Thank you!
- Fran
Capgemini
IN
Hi Fran,
I think the connect-soap is trying to connect more than one time resulting the long timeout.
If you trace in prod and the scenario is easily reproducible then life is easy.
if not then you need to turn logging on for the invokeAxis2 in connect-soap rule and see the timings. Again if SOAP is getting fired all the time in the application then it would create huge log.
Thanks
Saikat
Pegasystems Inc.
JP
Hello,
Thank you for the feedback, but you even stated that turning on enhanced logging for a heavily used interface, which this is, will bog down our logging in PROD, so I won't be able to use this approach. Do you have any other suggestions?
Why would the Connect behaviour be different in DEV and PROD?
Thanks for your continued help!
- Fran
Capgemini
IN
Yes logging invokeAxis might create lots of logs.
2 ways:
1.You can store all the invokeaxis logs in different file that normal pega logs. Cofiguring the prlogging.xml.
or. 2,. Try jvm logging.http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html
-Saikat
Accepted Solution
Capgemini
IN
My guess is connect rule from pega is not behaving differently in dev or prod. The service provider is behaving differently. They are taking good amount of time to return endpoint failure in prod. And then pega is trying multiple time to connect to the same endpoint resulting the timeout.
-Saikat