Question
AIG
US
Last activity: 15 Mar 2021 9:20 EDT
Queue processor stream are duplicating reprocessing of kafka stream at retention
We are using Queue processor methodology - dedicated queue for real time event push to reporting system. We didn't had duplciate issue with lagecy "connect queue" agent processing, but queue processor kafka we are seeing issue "same message is coming to processing again after couple of hours [even it had processed prior] ". Please help to resolve this duplicate execution every day we are seeing ~ 250 messages; looks like it happen at retention cleanup time.( I think so)
it hard to trace queue processor in 1.2 -1.5 million event and pattern i am seeing after 8 -12 hours at might duplication
J P Morgan Chase
IN
we have observed the similar issue where we have queued items being processed multiple times. Along with this issue , we noticed that the Stream node is going down and due to which the Queue Processor execution is failing - giving PEGA0072 - DataFlow.backgroundProcessing service is failed.
ING
NL
@BHUSHANPATIL Hi..Did you get a solution to this issue?
AIG
US
@INFANTJESUS yes we had bad stream node. stream node was renamed with web node but kafka was holding partition with it and impacting.
We have deleted kafaka_data folder to restore. GCS has ran couple of linux command to find partion which are not reachable or consuming.
Updated: 15 Mar 2021 9:20 EDT
Tata Consultancy Services
US
@BHUSHANPATIL We are also facing this issue where a dedicated queue processor with immediate processing which runs at 12 midnight ET is running multiple times. We have started seeing this issue after daylight time changes from EST to EDT. We process 100k items with these queue processor per day and duplicate processing of same queue processor item is causing a lot of issues.