Question
Mindtree Ltd
US
Last activity: 28 Apr 2022 6:35 EDT
Queue Processor Lock issue on retrial
We have a queue processor activity which advances the case using performFlowAction (OOTB) activity. The underlying flow action post processing activity makes call to external service and updates the case with service response. This is working fine when there are no failures. When there is a failure with service, the post processing activity sets error messages on work page which will be propagate up till the queue processor activity.
As there are error messages, queue processor considering this as a failure and moving the queue entry into DelayedItems and trying to process after 2 mins (retrial attempt interval).
When queue processor tries to process 2nd/3rd/4th/5th time, getting locking issues (Lock already held by another process). It seems like Queue Processor is not releasing lock when there are failures. Any insights would be helpful on this. The same activity and similar configuration used to work fine with Standard agent. Updated with the extracted debug logs for the usecase (Attached). From the logs,
We have a queue processor activity which advances the case using performFlowAction (OOTB) activity. The underlying flow action post processing activity makes call to external service and updates the case with service response. This is working fine when there are no failures. When there is a failure with service, the post processing activity sets error messages on work page which will be propagate up till the queue processor activity.
As there are error messages, queue processor considering this as a failure and moving the queue entry into DelayedItems and trying to process after 2 mins (retrial attempt interval).
When queue processor tries to process 2nd/3rd/4th/5th time, getting locking issues (Lock already held by another process). It seems like Queue Processor is not releasing lock when there are failures. Any insights would be helpful on this. The same activity and similar configuration used to work fine with Standard agent. Updated with the extracted debug logs for the usecase (Attached). From the logs,
- we can see Reqeustor BGDSN0FIVDWW7S9O8CI9XIRPOF277CDFBA tried to process the queue entry for CL-135004 at 2022-04-04T14:37:00 and requeued for retrial after 1 min due to save error.
- After 1 min, at 2022-04-04T14:38:10, another requestor BD60UAL7WGQX7KMRUCYG7G3OC8QQW5G9HA attempts to process the same case but failed to acquire the lock. And lock information shows that owner of the lock is the first requestor.
- lock information : {pxObjClass=System-Locks, pxLockHandle=MYORG-IDK-CL-WORK CL-135004, pxUpdateOpName=Queue processor(IDKQueueProcessor), pxUpdateOperator=System, ErrorMessage=Database-LockFailure-LockHeldByAnother, pxOwnerId=BGDSN0FIVDWW7S9O8CI9XIRPOF277CDFBA, pxExpireDateTime=20220404T150705.019 GMT}
Thanks.
***Edited by Moderator Marije to add Capability tags***
Eclatprime Digital Private Limited
IN
I am also facing the same issue. Did you find any solution ?
Mindtree Ltd
US
@PIYUSH SINHA I havent got any solution. As a work around, I explicitly used Page-Unlock to overcome this issue.
Pegasystems Inc.
GB
@SanthoshB6893 can you confirm the settings and configuration of your scenario? Please see the documentation 'Control the locking and retry mechanisms of background processes (8.3)' (Also see summary here)
If an error occurs during the processing of the message, then an entry will be written into the class System-Message-QueueProcessor-BrokenItem and records the message as broken-process in DB table. The message in the broken item class will have the actual failure reason which can be used for debugging purposes
If there is a lock failures or any exception, after the number of retries are exceeded the entry should go into Broken Queue. Is this occuring?
Have you tried to increase the value of delay factor in the QP as suggested in this post and this post?
There is also a suggestion to use Activity-Set-Status as fail so it will go for retrial, but the user was not sure whether it will stop after max attempts in Queue processor rule.
@SanthoshB6893 can you confirm the settings and configuration of your scenario? Please see the documentation 'Control the locking and retry mechanisms of background processes (8.3)' (Also see summary here)
If an error occurs during the processing of the message, then an entry will be written into the class System-Message-QueueProcessor-BrokenItem and records the message as broken-process in DB table. The message in the broken item class will have the actual failure reason which can be used for debugging purposes
If there is a lock failures or any exception, after the number of retries are exceeded the entry should go into Broken Queue. Is this occuring?
Have you tried to increase the value of delay factor in the QP as suggested in this post and this post?
There is also a suggestion to use Activity-Set-Status as fail so it will go for retrial, but the user was not sure whether it will stop after max attempts in Queue processor rule.
If it is not then increment integer property on primary page and don't set Activity-Set-Status if it crosses max attempts. See this post 'Creating Queue Processor: How the items are Requeued in queue processor'.
If your issue needs a more in-depth look then I would suggest that you log a support incident for this so that you can provide GCS with the full debug logs and examples.
Please provide us with the support ticket ID here so that we can help you track the resolution.
Mindtree Ltd
US
@MarijeSchillern I see retry attempts are being made by the Queue processors and entries are written to Delayed queue during retrial with incremental delays. After running for configured number of maximum times, entry is added to Broken queue. When there is an error and queue processor knows that there is a failure and adding an entry to delayed queue, why the lock is still hold by it for default interval (30 mins) is not clear to me. I have enabled debug logs and attached the same to the original question. For now to keep the functionality working with queue processors, I have used Page-Unlock method in the queue processor activity with which its working fine.
Lventur
IN
As far I know this is how pega product will work, Agent/Queue Processor will acquire case lock while processing request based on your configurations but during exception it seems you are not having logic to release case lock explicitly...
When some error is getting happened while queue processor/agent processing any request, please use Page-Unlock method and release the lock manually... So that when retry request is getting processed without any lock issue system will be able to process request..
Mindtree Ltd
US
@Gunasekaran Baskaran thank you for your response. But I expect the QP process to take care of the locking in case of errors. Agents does take care of locking in either of the cases, Success or failure. As I mentioned above, adding explicit Page-Unlocked solve the issue though. Thank you
Pegasystems Inc.
GB
@SanthoshB6893 Agent documentation and forum posts shows that Agents encountered similar issues and locks would cause failures.
As discussed above, likely there is a configuration that is preventing the QP from retrieving a released lock (eg code from an activity or an obj-save). This post offers advice on avoiding the scenarios.
You can find an interesting Discussion about the Pega lock mechanism here.
can you confirm that your question has been answered?