We have a requirement to process messages based on a key in sequence.
How to modify the partition key for a queue processor to consider a custom key to make sure the sequence is preserved.
***Edited by Moderator: Pooja Gadige to add capability tags***
***Edited by Moderator Marissa to change Type from Discussion to Question; add Idea tag***
***Edited by Moderator Marije to add existing FDBK ID tag***
The SME's responded by informing us that Pega Platform doesn’t support processing messages in an ordered sequence.
Kafka (QP) are designed for processing massive amounts of messages asynchronously. There was never enough demand for such capabilities that would justify implementing such a complex functionality in the product.
Currently there are no plans to implement this as error-situation handling put this out of scope.
The recommended OOTB method would be to exclude the use of QPs. The downside of this solution is that throughput is limited by the single processing thread. But this is more suitable and much less complex to implement and may therefore fall within support:
1. Manage your work in form of cases. Case ID is incremental so that defines our desired ordering.
2. Create a Job Scheduler with a cluster-level lock (such a JS will run only on a single node at a time). Schedule should depend on the amount of data that needs to be processed but it could run rather frequently (i.e. every 10s, 1m)
3. Have the JS pick up a single case from the top of the list and process it. Handle all processing errors within the JS implementation. After successful processing, mark the case as completed.