Question
Royal Bank of Canada
CA
Last activity: 20 Mar 2023 12:31 EDT
Queue processor, <> couldn't reach a stable state. Please check after some time.
Hi All,
Lately I am seeing this issue on our Dev system which has a single node configured as Stream, BackgroundProcessing. I created a dedicated queue processor and after processing some 1000 records it just went into Pause state. All queued items are showing up under Ready To Process and are not moving from there. When I checked the Data Flow it is showing state as "Pausing" and 100 warnings for stale threads detected. There was no failure and all 1000 records processed successfully.
When I tried to stop the queue it's throwing an error "Queue processor, <> couldn't reach a stable state. Please check after some time." I have no idea on how much time I can wait as this is in this state since yesterday. I checked the Stream service too and it's in Normal status with Action as Execute. Could some one please help me what might be the cause and how to prevent this happening in future? Will this be a severe issue on Production?
Accepted Solution
Updated: 20 Mar 2023 12:31 EDT
Royal Bank of Canada
CA
@KRISHNAPAVANP As I said I did the below changes and didn't see this issue for sometime. Later I was moved out of that work and so I too have no idea on what happened later. You can try below and see if it works for you.
Meanwhile I tried using the "None" option for "Lock using" and it looks like working without any issue. The queue never went down till now and it's fast too. So, you can try using "None" option with "Queue current snapshot of page" option in the Queue-For-Processing method configuration and see how it is working. I didn't enable "Queue current snapshot of page" till now but you can toggle this option and see how it performs. With this option enabled it should gain even more performance.
Royal Bank of Canada
CA
@Will Cho We raised an SR with Pega support and they are still looking into this. Yesterday we had a meeting with Pega support team and shared the documents and logs with them. I will let you know once we get a resolution from them.
Meanwhile I tried using the "None" option for "Lock using" and it looks like working without any issue. The queue never went down till now and it's fast too. So, you can try using "None" option with "Queue current snapshot of page" option in the Queue-For-Processing method configuration and see how it is working. I didn't enable "Queue current snapshot of page" till now but you can toggle this option and see how it performs. With this option enabled it should gain even more performance.
Thank you.
Pegasystems Inc.
US
Hi @DurgaPrasadBali,
Could you share the Support Case ID with us so that we can connect this post to it?
Thank you!
t
US
@Will Cho I had same issue. Later we found that the Queue processor activity have code issue which caused the activity to execute in a loop for hours. It caused the stability issue. if your activity executes for a long time, then see if optimizing the code fixes the issue (Alternatively change the default settings in Queue processor rule for long running Queue processor)
Updated: 20 Mar 2023 12:41 EDT
Pegasystems Inc.
GB
Sometimes these issues occur if one stream service node out of multiple nodes is not reachable. Solution is to recycle unreachable stream service node.
Please provide the below artifacts so we can investigate further.
1. Go to the Configure > Decisioning > Infrastructure > Services > Stream landing page, click on NORMAL and capture screenshots of each stream node metrics.
2. Download the Kafka diagnostic for each stream node (acquired from Stream landing page > click on each node > click on Show diagnostics)
3. Capture all Pega logs (RULES, ALERT, CLUSTER) from the stream nodes.
---> could you share the Support Case ID number?
If the issue is resolved could you mark a 'Accept Solution' so that we may close this post?
Please also confirm which Troubleshooting articles you have checked:
https://support.pega.com/question/queue-processor-not-working-after-upgrade-821-862
https://docs-previous.pega.com/system-administration/87/queue-processor-faq
https://docs-previous.pega.com/system-administration/87/tracing-background-processing
HealthFirst Org
US
@DurgaPrasadBali Is this issue resolved ?
We are also facing same issue.
Express scripts
US
@KRISHNAPAVANP Is this issue resolved? We are also facing same issue after upgrading from 8.2 to 8.8
Royal Bank of Canada
CA
As I said I did the below changes and didn't see this issue for sometime. Later I was moved out of that work and so I too have no idea on what happened later. You can try below and see if it works for you.
Meanwhile I tried using the "None" option for "Lock using" and it looks like working without any issue. The queue never went down till now and it's fast too. So, you can try using "None" option with "Queue current snapshot of page" option in the Queue-For-Processing method configuration and see how it is working. I didn't enable "Queue current snapshot of page" till now but you can toggle this option and see how it performs. With this option enabled it should gain even more performance.
Accepted Solution
Updated: 20 Mar 2023 12:31 EDT
Royal Bank of Canada
CA
@KRISHNAPAVANP As I said I did the below changes and didn't see this issue for sometime. Later I was moved out of that work and so I too have no idea on what happened later. You can try below and see if it works for you.
Meanwhile I tried using the "None" option for "Lock using" and it looks like working without any issue. The queue never went down till now and it's fast too. So, you can try using "None" option with "Queue current snapshot of page" option in the Queue-For-Processing method configuration and see how it is working. I didn't enable "Queue current snapshot of page" till now but you can toggle this option and see how it performs. With this option enabled it should gain even more performance.
Infosys
NL