Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 Sanivarapu Veera Vikram Reddy (SVeeraVikramReddy)
NA
Consultant
NA
IN
SVeeraVikramReddy Member since 2012 5 posts
NA
Posted: May 10, 2016
Last activity: May 25, 2016
Posted: 10 May 2016 0:09 EDT
Last activity: 25 May 2016 16:24 EDT
Closed

What could be the possible reason for thousands of Scheduled items remaining in the Agent Queue on daily basis (SR-A24141)?

Hi All,

In our application we have configured a standard agent as shown below. On a daily basis we will be queuing 10000+ instances into the agent queue. It has been observed that, the instances are getting piled up in the scheduled queue, and no items are neither being moved to broken queue. Agent is being shown as running successfully. From logs, it has been observed that, service calls in the agent activity are getting failed. This is the only widely seen error in the logs. In this case atleast the instances should be moved to broken queue, but still not happening. Did anyone of you have ever faced this scenario? Request you to kindly provide your suggestions. We are using Pega PRPC 7.1.8

Note: SR-A24141 has been raised for this issue. Service Engineer advised to post the issue in this forum, to get expert advice.

Agent:

Capture.JPG

Scheduled Items:

Show More

Hi All,

In our application we have configured a standard agent as shown below. On a daily basis we will be queuing 10000+ instances into the agent queue. It has been observed that, the instances are getting piled up in the scheduled queue, and no items are neither being moved to broken queue. Agent is being shown as running successfully. From logs, it has been observed that, service calls in the agent activity are getting failed. This is the only widely seen error in the logs. In this case atleast the instances should be moved to broken queue, but still not happening. Did anyone of you have ever faced this scenario? Request you to kindly provide your suggestions. We are using Pega PRPC 7.1.8

Note: SR-A24141 has been raised for this issue. Service Engineer advised to post the issue in this forum, to get expert advice.

Agent:

Capture.JPG

Scheduled Items:

Capture2.JPG

Thanks & Regards,

S.Veera Vikram Reddy


Show Less
To see attachments, please log in.
System Administration

Posted: 8 years ago
Posted: 10 May 2016 6:37 EDT
Santanu Bhattacherjee (Santanu) Senior Manager, Cloud Engineering, Management Plane Services
Pegasystems Inc.
IN

Hello

You said in the log, "service calls in the agent activity are getting failed ""

Can you please give more detail about the error you are seeing in the log.

If you can provide us the error snippet from log then it can help us to analyze further.

To see attachments, please log in.
Posted: 8 years ago
Posted: 10 May 2016 6:59 EDT
Sanivarapu Veera Vikram Reddy (SVeeraVikramReddy)
NA
Consultant
NA
IN

Hi Santanu,

Please find the below log with the error. I attached the full PegaRULES & Alert logs to the SR-A24141

2016-04-26 00:02:02,006 [.PRPCWorkManager : 6] [  STANDARD] [   ***********] (      internal.mgmt.Executable) ERROR Rule-Connect-SOAP.***********  - Exception

com.pega.pegarules.pub.services.RemoteApplicationException: SOAP service failed

at com.pegarules.generated.activity.ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.step15_circum0(ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.java:3994)

at com.pegarules.generated.activity.ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.perform(ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.java:308)

at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3500)

at com.pegarules.generated.activity.ra_action_invoke_c28ff8c27ac24045d30f5561679549d1.step8_circum0(ra_action_invoke_c28ff8c27ac24045d30f5561679549d1.java:1037)

at com.pegarules.generated.activity.ra_action_invoke_c28ff8c27ac24045d30f5561679549d1.perform(ra_action_invoke_c28ff8c27ac24045d30f5561679549d1.java:189)

at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3500)

Show More

Hi Santanu,

Please find the below log with the error. I attached the full PegaRULES & Alert logs to the SR-A24141

2016-04-26 00:02:02,006 [.PRPCWorkManager : 6] [  STANDARD] [   ***********] (      internal.mgmt.Executable) ERROR Rule-Connect-SOAP.***********  - Exception

com.pega.pegarules.pub.services.RemoteApplicationException: SOAP service failed

at com.pegarules.generated.activity.ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.step15_circum0(ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.java:3994)

at com.pegarules.generated.activity.ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.perform(ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.java:308)

at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3500)

at com.pegarules.generated.activity.ra_action_invoke_c28ff8c27ac24045d30f5561679549d1.step8_circum0(ra_action_invoke_c28ff8c27ac24045d30f5561679549d1.java:1037)

at com.pegarules.generated.activity.ra_action_invoke_c28ff8c27ac24045d30f5561679549d1.perform(ra_action_invoke_c28ff8c27ac24045d30f5561679549d1.java:189)

at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3500)

at com.pegarules.generated.activity.ra_action_******_256fcf1e9556cea99c69e4e87c09c7f3.step4_circum0(ra_action_getagentop_256fcf1e9556cea99c69e4e87c09c7f3.java:514)

at com.pegarules.generated.activity.ra_action__******__256fcf1e9556cea99c69e4e87c09c7f3.perform(ra_action_******_256fcf1e9556cea99c69e4e87c09c7f3.java:137)

at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3500)

at com.pega.pegarules.session.internal.mgmt.Executable.invokeActivity(Executable.java:10521)

at com.pegarules.generated.activity.ra_action__******__238f7a01a4ca276cbadc33e7ebbafcbc.step2_circum0(ra_action__******__238f7a01a4ca276cbadc33e7ebbafcbc.java:936)

at com.pegarules.generated.activity.ra_action__******__238f7a01a4ca276cbadc33e7ebbafcbc.perform(ra_action__******__238f7a01a4ca276cbadc33e7ebbafcbc.java:86)

at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3500)

at com.pega.pegarules.session.internal.mgmt.Executable.invokeActivity(Executable.java:10521)

at com.pegarules.generated.activity.ra_action__******__5aab21f92293cb443587e3bff0c46f62.step27_3_circum0(ra_action__******__5aab21f92293cb443587e3bff0c46f62.java:3229)

at com.pegarules.generated.activity.ra_action__******__5aab21f92293cb443587e3bff0c46f62.step27_0_circum0(ra_action__******__5aab21f92293cb443587e3bff0c46f62.java:3006)

at com.pegarules.generated.activity.ra_action__******__5aab21f92293cb443587e3bff0c46f62.perform(ra_action__******__5aab21f92293cb443587e3bff0c46f62.java:643)

at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3500)

at com.pega.pegarules.session.internal.mgmt.Executable.invokeActivity(Executable.java:10521)

at com.pegarules.generated.activity.ra_action__******__fd6932fb5c268e74d03fd95eeb5ec56f.step8_circum0(ra_action__******__fd6932fb5c268e74d03fd95eeb5ec56f.java:695)

at com.pegarules.generated.activity.ra_action__******__fd6932fb5c268e74d03fd95eeb5ec56f.perform(ra_action_mapantifraudproperties_fd6932fb5c268e74d03fd95eeb5ec56f.java:211)

at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3500)

at com.pega.pegarules.session.internal.mgmt.Executable.invokeActivity(Executable.java:10521)

at com.pegarules.generated.activity.ra_action__******__a698a6cf4e7694716bb30d04559488c8.step9_circum0(ra_action__******__a698a6cf4e7694716bb30d04559488c8.java:824)

at com.pegarules.generated.activity.ra_action__******__a698a6cf4e7694716bb30d04559488c8.perform(ra_action__******__a698a6cf4e7694716bb30d04559488c8.java:217)

at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3500)

at com.pega.pegarules.session.internal.async.agent.QueueProcessor.runActivity(QueueProcessor.java:721)

at com.pega.pegarules.session.internal.async.agent.QueueProcessor.execute(QueueProcessor.java:602)

at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:945)

at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1259)

at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1008)

at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:901)

at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:670)

at com.pega.pegarules.session.internal.async.Agent.queueBatchActivityForAgent(Agent.java:1504)

at com.pega.pegarules.session.internal.async.Agent.queueBatchActivityForAgent(Agent.java:1293)

at com.pega.pegarules.session.internal.async.AgentQueue.run(AgentQueue.java:792)

at com.pega.pegarules.session.internal.async.Agent.runAgentTask(Agent.java:936)

at com.pega.pegarules.session.internal.engineinterface.etier.impl.AsyncServant.processAsyncTask(AsyncServant.java:221)

at com.pega.pegarules.session.internal.engineinterface.etier.impl.AsyncServant.invoke(AsyncServant.java:167)

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:315)

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)

at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:225)

at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

at java.lang.reflect.Method.invoke(Method.java:611)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:367)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:408)

at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:223)

at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:245)

at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)

at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.invokeEngine(Unknown Source)

at com.pega.pegarules.session.internal.engineinterface.etier.mdb.PRAsync.processRequest(PRAsync.java:155)

at com.pega.pegarules.session.internal.engineinterface.etier.asynchbeans.AsyncTask.run(AsyncTask.java:64)

at com.pega.pegarules.session.internal.engineinterface.etier.asynchbeans.RunnableWrapper.run(RunnableWrapper.java:71)

at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:268)

at java.security.AccessController.doPrivileged(AccessController.java:273)

at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:768)

at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:222)

at com.ibm.ws.asynchbeans.ABWorkItemImpl.run(ABWorkItemImpl.java:159)

at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)

Caused by:

com.pega.apache.axis2.AxisFault: Exception occured while processing the request

at com.pega.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)

at com.pega.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:375)

at com.pega.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:421)

at com.pega.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)

at com.pega.apache.axis2.client.OperationClient.execute(OperationClient.java:165)

at com.pegarules.generated.activity.ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.step14_circum0(ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.java:3891)

at com.pegarules.generated.activity.ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.perform(ra_action_invokeaxis2_e0a499f26a95053ac6300b525bd7d41f.java:291)

... 58 more


Show Less
To see attachments, please log in.
Posted: 8 years ago
Posted: 10 May 2016 6:44 EDT
Santanu Bhattacherjee (Santanu) Senior Manager, Cloud Engineering, Management Plane Services
Pegasystems Inc.
IN

Also, it is a good idea to try tracing the agent .

You can review this PegaTube video to learn the technique on how to trace an agent, : https://collaborate.pega.com/discussion/video-support-guide-how-trace-agent-designer-studio

To see attachments, please log in.
Posted: 8 years ago
Posted: 10 May 2016 9:43 EDT
Santanu Bhattacherjee (Santanu) Senior Manager, Cloud Engineering, Management Plane Services
Pegasystems Inc.
IN

Thanks. Since there is already an SR I would leave this to the engineer who is working on this issue to take forward.

Meanwhile can you please update your post with the SR number

To see attachments, please log in.
Posted: 8 years ago
Posted: 10 May 2016 23:56 EDT
Ravikiran C (cr001)
Pegasystems Inc.
Senior Technical Solutions Engineer
Pegasystems Inc.
IN

Hi Santanu,

I have shared him the below document on how the queueing will be done for the standard agents. In this case even though the exception has been entered in the log file for the SOAP call which is happening in the Agent Activity the unsuccessfully processed items are not getting moved into the broken queue.

https://docs-previous.pega.com/system-administration/how-agent-queues-work

Veera has informed that he will remove the SOAP calls from the Activity and will check. He is in a process of doing that. He is very curious to understand the actual working of the Standard Agent queueing mechanism practically instead of going through the document. Which is why i have suggested him to raise a community discussion for more exposure to the expertise like you.

Please let me know if you have any inputs for this.

Regards,

Ravi

To see attachments, please log in.
Posted: 8 years ago
Posted: 11 May 2016 0:32 EDT
Ravikiran C (cr001)
Pegasystems Inc.
Senior Technical Solutions Engineer
Pegasystems Inc.
IN

Hi Santanu,

One more thing i would like to add , as we know that the queued items will be processed in First in First out (FIFO). Veera would like to know is there any condition where this FIFO order would be skipped or not followed.

Please share your thoughts.

Regards,

Ravi

To see attachments, please log in.
Posted: 8 years ago
Posted: 16 May 2016 4:22 EDT
Santanu Bhattacherjee (Santanu) Senior Manager, Cloud Engineering, Management Plane Services
Pegasystems Inc.
IN

Hi Ravikiran,

For your queries :

1. He is very curious to understand the actual working of the Standard Agent queueing mechanism practically instead of going through the document

>> I believe , in order to understand the mechanism practically one needs to go thru the courses available in Pega Academy which generally comes with hands on example.

2. Veera would like to know is there any condition where this FIFO order would be skipped or not followed.

>> I am not aware of any such mechanism thru which you can skip the existing FIFO order .If you are using queue for agent method then probably playing with this can give some clue "MinimumAgeforProcessing "parameter. But not sure.

Others can comment.

To see attachments, please log in.
Posted: 8 years ago
Posted: 20 May 2016 9:42 EDT
Sumit Kumar Kundan (Sumit_PEGA)
Rabobank
Sumit Kumar Kundan
Rabobank
NL

I am thinking in some way.

If on a particular node where agent is enabled. But stopped in SMA.

So in that case agent will pick the item but attempt will be 0.

Can you check whether agent is active on all nodes in SMA.

To see attachments, please log in.
Posted: 8 years ago
Posted: 25 May 2016 16:24 EDT
Don Zhang (ZHAND)
PEGA
Principal Service Assurance Advisor
Pegasystems Inc.
HK

From the screen shot, there is no Class and Activity specified.

For Best Practice, also specify the Category dropdown.

Better to find another Queue to test your Agent b/c current Queue has too many items.

To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice