Background agent is still running even if it got switched off, also it is processing the next item from the queue right after finishing the previous not caring about recurring time
Hi all,
I need some help to understand a strange behavior by running a standard auto queued agent.
We have a standard auto queued agent which calls an activity and does some work in a periodic time of 120 sec.
This agent was already running in production quite well for weeks.
Hi all,
I need some help to understand a strange behavior by running a standard auto queued agent.
We have a standard auto queued agent which calls an activity and does some work in a periodic time of 120 sec.
This agent was already running in production quite well for weeks.
Now we added a new step at the beginning to re-queue the current work item for minimum process time of 24h based on a condition and write an InfoForce log to see whether this items got re-queued.
After we had deployed this agent within a new ruleset version, we faced the following in the logs:
2017-12-09 10:18:42,541 [.PRPCWorkManager : 7] [ STANDARD] [ ] [ Ruleset:02.01.01] (Mgmt_TransferOfContacts.Action) INFO - Skipped
2017-12-09 10:18:42,638 [.PRPCWorkManager : 7] [ STANDARD] [ ] [ Ruleset:02.01.01] (Mgmt_TransferOfContacts.Action) INFO - Skipped
2017-12-09 10:18:42,746 [.PRPCWorkManager : 7] [ STANDARD] [ ] [ Ruleset:02.01.01] (Mgmt_TransferOfContacts.Action) INFO - Skipped
2017-12-09 10:18:42,811 [.PRPCWorkManager : 7] [ STANDARD] [ ] [ Ruleset:02.01.01] (Mgmt_TransferOfContacts.Action) INFO - Skipped
2017-12-09 10:18:42,876 [.PRPCWorkManager : 7] [ STANDARD] [ ] [ Ruleset:02.01.01] (Mgmt_TransferOfContacts.Action) INFO - Skipped
2017-12-09 10:18:42,946 [.PRPCWorkManager : 7] [ STANDARD] [ ] [ Ruleset:02.01.01] (Mgmt_TransferOfContacts.Action) INFO - Skipped
2017-12-09 10:18:43,037 [.PRPCWorkManager : 7] [ STANDARD] [ ] [ Ruleset:02.01.01] (Mgmt_TransferOfContacts.Action) INFO - Skipped
...and a lot of more logs like this...
How this can happen? The Agent is setup to recurring each 120sec?
After a while the condition in the first step changed and the work items got processed one by one again without the recurring time of 120sec.
This brought us another issue: OutOfMemoryError!
I can imagine this because of thousands of items which were scheduled and the agent where running them without waiting the time setup for recurring; maybe even in the same Thread!?
If the requestor run all items really within the same thread, which I didn't thought it should work like, the pages where not cleaned up properly from the engine and would cause for sure an OutOfMemoryError after a while. The agent where processing about 1500 work items in 13h before running into an out of memory.
So, when this agent started to run again after restarting the node, we tried to switch off this agent in the SMA, but the agent doesn't stop at all and picked up next work items one by one which were scheduled, again without waiting 120sec like the agent scheduler has been setup.
Only when I interrupted the requestor in the SMA, the agent processing got stopped and ALL scheduled items in the queue (SYSTEM-QUEUE-DEFAULTENTRY) for the specific agent name got moved into the Broken-Queue.
Accepted Solution
USU GmbH
DE
Here is the asnwere of my last question: Why all scheduled 1500 items in the queue got moved into the broken queue by interrupting the current batch reqeustor for the running agent.
The upper pega help text is for older pega version 7.2.
But, our customer is using 7.2.2 and it seems there has been made a change in what is the default max record value.
Here is the pega help for max record in version pega 7.2.2.
For agents of the Standard queue mode, specify the maximum number of items from the agent queue that are processed at one time by the agent before it goes back to sleep for its interval.
If this field is blank, the default value is 0, and the agent processes all entries in its queue before sleeping.
EPAM Systems, Inc.
ES
Just to confirm, if its multi-node environment.
Please verify if the agent is running on any other nodes?
https://collaborate.pega.com/comment/120496 It will help you understand the behaviour of Auto-Queue Management.
Hope it helps you.
USU GmbH
DE
Yes, it is a multi-node-environment but the agent was running only on one node.
Thanks for the link. That is exactly why this agent is in standard mode to enable the advantages of AQM.
There are two steps which calls Queue-For-Agent.
1. In second step, If the process must be re-queued immediately
2. At the end of the process, if it has to be postponed and has to run a second time due to some conditions.
Pegasystems Inc.
IN
Hi,
Select a Broken-Process in System Queue Management and click on view it will show you the details about agent and exception if any .
Thanks,
Arun
USU GmbH
DE
Hi Arun,
all items have the same error message "PRInterruptedException".
That is why I interrupted the requestor to get the agent processing stopped, as this agent doesn't stop processing scheduled items by switching it off.
I checked all available agent scheduler on all 5 nodes and the agent was switched off for sure but still processing scheduled work objects.
USU GmbH
DE
I found the cause why the agent picked up so many in one single session.
Unfortunately, I missed this information which I found now in pega help:
I found the cause why the agent picked up so many in one single session.
Unfortunately, I missed this information which I found now in pega help:
| Max Records |
For agents of the This value is the maximum number of items to process. If Max Records is 30 and there are only 20 entries in the agent queue when the agent wakes, the agent processes the 20 entries and sleeps; it does not wait for an additional 10 entries. |
As we left the max record filed empty for the standard agent in the agent rule, up to 50 scheduled items got processed in on single session because there were thousand items in the queue.
--
But I still don't understand why all 1500 scheduled items got moved into the broken queue when I interrupt the requestor and not only 50 which would explain the default max record size.
Is there an option to change the max recode default value?
Accepted Solution
USU GmbH
DE
Here is the asnwere of my last question: Why all scheduled 1500 items in the queue got moved into the broken queue by interrupting the current batch reqeustor for the running agent.
The upper pega help text is for older pega version 7.2.
But, our customer is using 7.2.2 and it seems there has been made a change in what is the default max record value.
Here is the pega help for max record in version pega 7.2.2.
For agents of the Standard queue mode, specify the maximum number of items from the agent queue that are processed at one time by the agent before it goes back to sleep for its interval.
If this field is blank, the default value is 0, and the agent processes all entries in its queue before sleeping.