Discussion
Morgan Stanley
IN
Last activity: 9 Jan 2023 7:48 EST
Change kafka partition key for a queue processor
We have a requirement to process messages based on a key in sequence.
How to modify the partition key for a queue processor to consider a custom key to make sure the sequence is preserved.
***Edited by Moderator Marissa to change Type from Discussion to Question; add Idea tag***
***Edited by Moderator Marije to add existing FDBK ID tag***
Updated: 3 Jan 2023 12:20 EST
Infosys
AU
This would be a really great addition to the QP config.
I second this feature request..
Updated: 3 Jan 2023 12:20 EST
Morgan Stanley
IN
@Avinash.Haridasu @MarissaRogers Can this be taken as an enhancement request.
Pegasystems Inc.
US
Yes, I will move this to a Discussion type post and tag it with Idea.
Pegasystems Inc.
GB
The SME's responded by informing us that Pega Platform doesn’t support processing messages in an ordered sequence.
Kafka (QP) are designed for processing massive amounts of messages asynchronously. There was never enough demand for such capabilities that would justify implementing such a complex functionality in the product.
Currently there are no plans to implement this as error-situation handling put this out of scope.
The recommended OOTB method would be to exclude the use of QPs. The downside of this solution is that throughput is limited by the single processing thread. But this is more suitable and much less complex to implement and may therefore fall within support:
1. Manage your work in form of cases. Case ID is incremental so that defines our desired ordering. 2. Create a Job Scheduler with a cluster-level lock (such a JS will run only on a single node at a time). Schedule should depend on the amount of data that needs to be processed but it could run rather frequently (i.e. every 10s, 1m) 3. Have the JS pick up a single case from the top of the list and process it. Handle all processing errors within the JS implementation. After successful processing, mark the case as completed.