Question
Wells Fargo
IN
Last activity: 3 Nov 2022 15:28 EDT
OOTB pzDelayedQueueProcessorSchedule Job scheduler getting stale , unless server is restarted
We have a requirement for which we used queue processor. The configurations done for the queue processor are:
In queue processor:
Type is selected as delayed
In Queue-For-Processing method,
1. Write now is not selected
2. Queue type is dedicated
3. Date/Time delay is given as 2 min from creation time
The items is getting queued and since it is delayed type of item , the pzDelayedQueueProcessorSchedule job scheduler is responsible for picked up the delayed item once the date/time selected while queuing is crossed. But in multiple instances and different environments, this OOTB job scheduler is going stale and the next run time is getting stuck at a certian date/time, way past the current time. Unless we restart the server we are unable to start the job scheduler back.
THis is causing the items which are delayed just getting stuck in the queue,.
***Edited by Moderator Marije to add Support Case Details; ***
Common Wealth Bank
AU
@SiriRamacharlu is this issue resolved? Restart the server each time is a workaround. Please share if you have received a hotfix for this issue?
Pegasystems Inc.
GB
@Ahmed462 I checked whether a support ticket was logged for this and I believe I identified the relevant investigation.
In the past we have seen issues like this occur when the hazelcast distributed lock held by pzDelayedQueueProcessorSchedule job scheduler is never released. This would then stop the future invocation of the Job Scheduler. The issue can occur the node where the job scheduler was running was either restarted while lock was still held at cluster level or the node itself has gone down or went out of cluster due to reasons like OOM, high CPU etc. This job scheduler is responsible for processing of all delayed items from the table and posting it to Ready To Process for a queue processor. As it is in locked state, all the items queued in the scheduled column are not getting picked for processing. We can clear the distributed lock either by a full cluster shutdown followed by a restart. However, this solution is not ideal for production environment. Due to that reason, Pega have also come-up with an utility which can help in clearing of the distributed locks from the hazelcast cluster whenever you run into the issue. You would need to log a support incident with our Global Client Support team for this.
@Ahmed462 I checked whether a support ticket was logged for this and I believe I identified the relevant investigation.
In the past we have seen issues like this occur when the hazelcast distributed lock held by pzDelayedQueueProcessorSchedule job scheduler is never released. This would then stop the future invocation of the Job Scheduler. The issue can occur the node where the job scheduler was running was either restarted while lock was still held at cluster level or the node itself has gone down or went out of cluster due to reasons like OOM, high CPU etc. This job scheduler is responsible for processing of all delayed items from the table and posting it to Ready To Process for a queue processor. As it is in locked state, all the items queued in the scheduled column are not getting picked for processing. We can clear the distributed lock either by a full cluster shutdown followed by a restart. However, this solution is not ideal for production environment. Due to that reason, Pega have also come-up with an utility which can help in clearing of the distributed locks from the hazelcast cluster whenever you run into the issue. You would need to log a support incident with our Global Client Support team for this.
User are usually asked to re-run the background process utility and restart the corresponding nodes. If the job is still locked, we would ask you to log a support incident and you provide the latest logs from startup of the node