Question
American Express
US
Last activity: 12 Jan 2016 10:05 EST
What's the Difference in log level setting for Soap Faults Exceptions in Pega6.2 & Pega7
HI Team,
We are getting lot of soap failure Exception in our production log files after Pega7 Upgrade, most of these failures are valid scenarios. Due to this log file size is getting huge. This behavior is not see in Pega 6.2, Can you please help us to achieve same behavior as Pega 6.2.
Scenario:
When we hit External system for particular service, if it sends Fail message in response automatically PEGA activity ‘InvokeAxis2’ handles SOAP fault message content and writes it in the log file which is used for debugging purpose(i.e.., OOTB Functionality) . When we trace same scenario in 6.2 we got the similar fail message for the ‘InvokeAxis2’ activity but the fail message is not getting logged in the log file.
Exception Name :
RemoteApplicationException : SOAP service failed
Caused by:
com.pega.apache.axis2.AxisFault: No Data found in DB
Valid Soap Faults :
- No Data Found in DB (Error Code : 500)
- Not valid Transactions
Analysis:
As an investigation I have compared the prlogging.xml files for Pega6.2 and Pega7 both are same.
Query's:
So I would like to known from you,
HI Team,
We are getting lot of soap failure Exception in our production log files after Pega7 Upgrade, most of these failures are valid scenarios. Due to this log file size is getting huge. This behavior is not see in Pega 6.2, Can you please help us to achieve same behavior as Pega 6.2.
Scenario:
When we hit External system for particular service, if it sends Fail message in response automatically PEGA activity ‘InvokeAxis2’ handles SOAP fault message content and writes it in the log file which is used for debugging purpose(i.e.., OOTB Functionality) . When we trace same scenario in 6.2 we got the similar fail message for the ‘InvokeAxis2’ activity but the fail message is not getting logged in the log file.
Exception Name :
RemoteApplicationException : SOAP service failed
Caused by:
com.pega.apache.axis2.AxisFault: No Data found in DB
Valid Soap Faults :
- No Data Found in DB (Error Code : 500)
- Not valid Transactions
Analysis:
As an investigation I have compared the prlogging.xml files for Pega6.2 and Pega7 both are same.
Query's:
So I would like to known from you,
Are there any new enhancements in Pega7 with respective Soap Faults Loggers ?
Why Soap Fails are getting logged only in Pega7 not in Pega6.2 ?
What shall we do to achieve same behaviour as Pega6.2 ?
Regards,
Manoj
Message was edited by: Kiran Radha Manoj Turaga
Accepted Solution
SR-A11492 was opened for this and solution is to modify invokeAxis2 step # 15 code
Throwable cause = e.getCause();
if (cause != null && cause instanceof java.io.IOException)
throw new ResourceUnavailableException("SOAP service failed", cause);
else
throw new RemoteApplicationException("SOAP service failed", e);
}
and replace
throw new RemoteApplicationException("SOAP service failed", e);
with
oLog.error("SOAP service failed", e);
Updated: 26 Oct 2015 10:44 EDT
There are a number of enhancements to prevent these types of "silent error" scenarios from occuring . If something is failing in the tracer, that error should be visible in the logfile as well. Silent errors are extremely difficult to diagnose without turning on Debug settings, which has a huge impact on logfile size and may even be completely infeasible to enable in a production setting. Silent errors/silent failures are generally considered a bug rather than an enhancement for this reason.
Are you rolling your logfile on a daily basis? How much space are each of your daily logs taking? Unless this is occuring constantly, I wouldn't expect the impact on logfile size to be a significant problem.
American Express
US
Hi Syl,
Can you please Elaborate this 'Silent Error' ? Do You mean silent failures are the bugs in 6.2 version which are corrected as part of Pega7 Upgrade Code ?
Yes we have logs rolled out on daily basis.
Before Pega 7 Upgrade our daily log file size is around 65 MB, After Pega 7 Upgrade our daily log file size is around 130 MB.
It has been doubled. Daily we are getting around 4000 errors on these Soap Failures. So can you please suggest us a solution for this issue.
If we turn off the log settings it will impact us when we really get an issue, we will not be able to trace it out.
Regards,
Manoj
Correct. As you mentioned, the SOAP service was failing silently in 6.2 and this has been corrected.
If you need to diagnose and/or address the root cause of the problem behind these SOAP errors, I would strongly recommend opening an SR via My Support Portal.
Are these SOAP requests being made by an external entity? Are you implying that these failures are caused by "bad requests" made by that external service?
American Express
US
Thanks Syl,
We normally hit External Service for different purposes, we will validate before sending the request. When the request is valid but the results are Zero then the third party will send us Fail message : NO Data Found in DB
So when ever Connect-Soap Receives Fail message it is logged by the Exception handling mechanism present in Pega Default Activity 'InvokeAxis2’.
But due to Pega 6.2 silent issue we are not able to see these errors in Pega 6.2 Log files
Where as in Pega7 we are able to see this in the log file, due to this log file size is huge. So our client is asking why this is happening in only Pega7 we never see this in Pega 6.2 as log file size is huge they cannot maintain it.
So would you suggest me to raise an SR with Pega Team ?
Regards,
Manoj
Hi Kiran,
I recommend turning on Wire debug for detecting SOAP Communication issues - see the video that demonstrates the troubleshooting approach here:
https://collaborate.pega.com/question/error-while-running-bat-file
American Express
US
Thanks Vipin,
We know why this soap error is getting triggered : There is due to no Data in DB for our request. But the external system is sending fail message as response.
But we don't have control over the pega activity, InvokeAxis2 is doing the job for us i.e.., writing the log file with the soap fail : Reason No data found in DB.
Which it didn't do this job in pega 6.2 but this is filling the logs in pega 7. This is our issue.
Can you please suggest : Syl & Vipin
Regards,
Manoj
Does the external service throw an error/exception or returns a SOAP fault? The correct behavior would be to return a soap fault which can then be processed by your calling activity appropriately.
American Express
US
Yes Vipin the external service is returning Exception
'com.pega.pegarules.pub.services.RemoteApplicationException: SOAP service failed'
Caused by:
com.pega.apache.axis2.AxisFault: No Data found in DB
at com.pega.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:446)
Yes we are handling appropriately according to the response given by the external system, but we don't have capability to remove the 'com.pega.pegarules.pub.services.RemoteApplicationException: SOAP service failed' Exception logged by Pega Default activity 'InvokeAxis2'. Can you please suggest us as this behavior is not observed in Pega 6.2 for the same scenario.
Regards,
Manoj
American Express
US
Vipin,
I have attached the stack trace to the 1st post attachments : Name - XXX Service PEGA7 and PEGA6.2 Log Comparision123
Regards,
Manoj
American Express
US
Vipin,
Have you verified the attached stack trace, is it the same as we discussed in the above thread(i.e.., Silent Bugs in Pega 6.2).
Regards,
Manoj
thanks for the stack trace. You can see that the exception stack is printed by step # 15 of invokeAxis2 Activity that Handles SOAP Fault:
com.pega.pegarules.pub.services.RemoteApplicationException: SOAP service failed
at com.pegarules.generated.activity.ra_action_invokeaxis2_c7367a7b950d78597cad7f60b4e360bf.step15_circum0(ra_action_invokeaxis2_c7367a7b950d78597cad7f60b4e360bf.java:4049)
if you look at the actual java code, it does this:
Throwable cause = e.getCause();
if (cause != null && cause instanceof java.io.IOException)
throw new ResourceUnavailableException("SOAP service failed", cause);
else
throw new RemoteApplicationException("SOAP service failed", e);
}
Does the java code look different in 6.2? Try turning debug on Rule_Obj_Activity.InvokeAxis2.Rule_Connect_SOAP.Action and that will tell you which step prints the failure message in the log.
Accepted Solution
SR-A11492 was opened for this and solution is to modify invokeAxis2 step # 15 code
Throwable cause = e.getCause();
if (cause != null && cause instanceof java.io.IOException)
throw new ResourceUnavailableException("SOAP service failed", cause);
else
throw new RemoteApplicationException("SOAP service failed", e);
}
and replace
throw new RemoteApplicationException("SOAP service failed", e);
with
oLog.error("SOAP service failed", e);
Pegasystems Inc.
US
Hi Vipin Kumar
Thanks for the SR. I have notated the thread with it for tracking!
Is this enhancement going into the code base? Treating soap faults as messages rather than exceptions should be a standard behavior, not something that requires a field modification to PRPC and an endless maintenance cycle as the change must be re-applied after any upgrade which changes connect-soap. The connect-soap rule should have an option, which defaults to false, for 'treat soap fault as an exception'