Question
Coforge DPA UK Ltd.
IN
Last activity: 18 Dec 2017 8:33 EST
Pega marketing failed due to error "Could not access HTTP invoker remote service VBD"
Hi,
There is an outbound marketing campaign in our application. It got failed during offer flow execution. have placed the complete track trace below. we are using a dedicated server for ADM and VBD. it is being accessed while running the campaign.
I have gone through some of the PDN support questions but not able to come to a conclusion after going through them.
I want to know the root cause of this issue and solution that I can apply for.
Error stack trace:
Hi,
There is an outbound marketing campaign in our application. It got failed during offer flow execution. have placed the complete track trace below. we are using a dedicated server for ADM and VBD. it is being accessed while running the campaign.
I have gone through some of the PDN support questions but not able to come to a conclusion after going through them.
I want to know the root cause of this issue and solution that I can apply for.
Error stack trace:
Error on node [<node ID here>] com.pega.dsm.dnode.api.dataflow.StageException: Exception in stage: Offer Flow at com.pega.dsm.dnode.api.dataflow.DataFlowStage$StageOutputSubscriber.onError(DataFlowStage.java:394) at com.pega.dsm.dnode.api.dataflow.DataFlowStageProcessor.onError(DataFlowStageProcessor.java:48) at com.pega.dsm.dnode.api.dataflow.DataFlowStage$StageInputSubscriber.onError(DataFlowStage.java:286) at com.pega.dsm.dnode.api.dataflow.DataFlowExecutor$QueueBasedDataFlowExecutor$3.process(DataFlowExecutor.java:202) at com.pega.dsm.dnode.api.dataflow.DataFlowExecutor$QueueBasedDataFlowExecutor.runEventLoop(DataFlowExecutor.java:119) at com.pega.dsm.dnode.api.dataflow.DataFlow$1.emit(DataFlow.java:162) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:56) at com.pega.dsm.dnode.impl.dataflow.manager.LocalRun.execute(LocalRun.java:145) at com.pega.dsm.dnode.impl.dataflow.manager.LocalRun.lockAndRun(LocalRun.java:86) at com.pega.dsm.dnode.impl.dataflow.manager.LocalRun.run(LocalRun.java:63) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:81) Caused by: com.pega.pegarules.pub.PRRuntimeException: Exception during data set execution at com.pegarules.generated.dataset.ra_action_actuals_d6f9b263cc16b7fa148ddb06161ef138.perform(ra_action_actuals_d6f9b263cc16b7fa148ddb06161ef138.java:123) at com.pega.pegarules.session.internal.mgmt.Executable.doAction(Executable.java:3310) at com.pegarules.generated.activity.ra_action_initiatebatchoffer_dafe47fbe294811a1dcc25c988d56f3a.step10_circum0(ra_action_initiatebatchoffer_dafe47fbe294811a1dcc25c988d56f3a.java:1603) at com.pegarules.generated.activity.ra_action_initiatebatchoffer_dafe47fbe294811a1dcc25c988d56f3a.perform(ra_action_initiatebatchoffer_dafe47fbe294811a1dcc25c988d56f3a.java:264) at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3505) at com.pegarules.generated.testgen.Rule_Decision_DDF_<segmentname here>_DFbd55d97dc5238f710955884aebe5463e9804ec28_Action_20170802T184857_858_GMT$4$1.onNext(Rule_Decision_DDF_segmentname here>_DFbd55d97dc5238f710955884aebe5463e9804ec28_Action_20170802T184857_858_GMT.java:235) at com.pega.dsm.dnode.api.dataflow.DataFlowStageBatchProcessor.commitBatchInternal(DataFlowStageBatchProcessor.java:102) at com.pega.dsm.dnode.api.dataflow.DataFlowStageBatchProcessor.commitBatch(DataFlowStageBatchProcessor.java:92) at com.pega.dsm.dnode.api.dataflow.DataFlowStageBatchProcessor.onPulse(DataFlowStageBatchProcessor.java:72) at com.pega.dsm.dnode.api.dataflow.DataFlowStage$StageInputSubscriber.onPulse(DataFlowStage.java:320) at com.pega.dsm.dnode.api.dataflow.DataFlowExecutor$QueueBasedDataFlowExecutor.runEventLoop(DataFlowExecutor.java:132) ... 8 more Caused by: org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [http:/<servername here>:8080/vbd/service/vbdService]; nested exception is java.net.SocketTimeoutException: Read timed out at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:216) at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:147) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) at com.sun.proxy.$Proxy51.insertFactRecords(Unknown Source) at com.pegarules.generated.pxInsertRecordsFromPages_071017_cBXTCjrCfGalvt_3BxEN8w.pxInsertRecordsFromPages07_10_17(pxInsertRecordsFromPages_071017_cBXTCjrCfGalvt_3BxEN8w.java:200) at com.pegarules.generated.pxInsertRecordsFromPages_071017_cBXTCjrCfGalvt_3BxEN8w.invoke(pxInsertRecordsFromPages_071017_cBXTCjrCfGalvt_3BxEN8w.java:91) at com.pega.pegarules.generation.internal.library.LibraryRuntime.resolveAndinvokeFunctionViaReflection(LibraryRuntime.java:133) at com.pega.pegarules.generation.internal.library.LibraryRuntime.invokeLibraryRuntime(LibraryRuntime.java:110) at com.pega.pegarules.session.internal.mgmt.Executable.invokeLibraryRuntime(Executable.java:9000) at com.pega.pegarules.priv.generator.LibrarySupport.resolveAndInvokeFunctionViaReflection(LibrarySupport.java:181) at com.pegarules.generated.pega_decisionengine_vbd.pxInsertRecordsFromPages(pega_decisionengine_vbd.java:345) at com.pegarules.generated.dataset.ra_action_actuals_d6f9b263cc16b7fa148ddb06161ef138$1$1.emit(ra_action_actuals_d6f9b263cc16b7fa148ddb06161ef138.java:152) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:320) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:52) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:98) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.toList(DataObservableImpl.java:77) at com.pegarules.generated.dataset.ra_action_actuals_d6f9b263cc16b7fa148ddb06161ef138.perform(ra_action_actuals_d6f9b263cc16b7fa148ddb06161ef138.java:93) ... 18 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) at com.pega.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) at com.pega.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) at com.pega.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) at com.pega.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1374) at com.pega.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1833) at com.pega.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1591) at com.pega.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:996) at com.pega.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) at com.pega.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at com.pega.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at com.pega.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:325) at com.pega.decision.util.spring.IgnorantHttpInvokerRequestExecutor.executePostMethod(IgnorantHttpInvokerRequestExecutor.java:250) at com.pega.decision.util.spring.IgnorantHttpInvokerRequestExecutor.doExecuteRequest(IgnorantHttpInvokerRequestExecutor.java:168) at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:138) at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:194) at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:176) at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:144) ... 34 more
thanks in advance.
Regards,
Durgarao J
Pegasystems Inc.
AU
Hi,
Is the Campaign run failing immediately after execution starts or after some time?
Which version of Pega and Marketing is in use?
Was this Campaign running without any issues earlier?
Is it a multinode environment or single node instance? Is Data Partitioning enabled?
Coforge DPA UK Ltd.
IN
Q: Is the Campaign run failing immediately after execution starts or after some time?
A: Campaign is failed after 20minutes.
Q: Which version of Pega and Marketing is in use?
A: Pega-7.1.9 & Marketing -7.13
Q: Was this Campaign running without any issues earlier?
A: Yes, it was running without any issue
Q: Is it a multinode environment or single node instance? Is Data Partitioning enabled?
A: It is a multinode environment. Data partitioning is enabled.
Q: Is the Campaign run failing immediately after execution starts or after some time?
A: Campaign is failed after 20minutes.
Q: Which version of Pega and Marketing is in use?
A: Pega-7.1.9 & Marketing -7.13
Q: Was this Campaign running without any issues earlier?
A: Yes, it was running without any issue
Q: Is it a multinode environment or single node instance? Is Data Partitioning enabled?
A: It is a multinode environment. Data partitioning is enabled.
Note: select * from app_pega_data.pr_data_vbd_config;
vbd.datasource.eviction.timeout=604800000.
Pegasystems Inc.
AU
Hi,
I could notice "SocketTimeoutException: Read timed out" in the log snippet, which might be due to the load.
Can you try reducing the Batch size and verify the behavior? Available under Marketing Portal --> Configuration --> Settings --> Application Settings.
Hope this helps!
Cheers,
Santhosh
Coforge DPA UK Ltd.
IN
Hi Santosh,
Thanks for your time and efforts.
Actually this issue was occured once and not able to replicate it again. So, there is no failure to test the suggestion you mentioned in the post,and there was no such failure earlier. batch size not created any issue earlier too.
Regards,
Durgarao J