Enforce First in First Out (FIFO) on Dedicated Queue Processors for specific case
Our application receives a lot of asynchronous updates on cases from external applications. We have dedicated Queue Processors with multiple partitions (20) for each to process these updates. Previously the same was processed by Standard Agents. As the standard agent uses a DB table to process the updates it naturally does FIFO. However for queue processors updates from the same case may go into different partitions and are being processed in a random order. From business point of view we would need all updates to be processed in sequence. We cannot reduce number of partitions to 1 as that would greatly reduce the throughput. Is there a way to set a Key on the message being posted on the Kafka topic for the queue processor, so that all updates for a particular case go to the same partition and therefore be processed in FIFO order?
Accepted Solution
Tata Consultancy Services
AU
@LAHIRIG1 - FIFO processing in Queue Processors is not available OOTB and is being considered as an enhancement for future release. Detailed explanation from PEGA engineer -
Issue primary reason: Pega - Enhancement Issue primary reason description:
Pega OOTB is trying to add this sequential processing feature in the Infinity 24 version.
Explanation type: Answer/Information Provided Explanation description:
With the current design, multiple partitions are used for QueueProcessor, and Queue-For-Processing pushes the item to any partition and order is not guaranteed OOTB as QP design is intended for high throughput but not sequential.
Order of the processing is not guaranteed with a queue processor as the queued messages are pushed to any partition and parallel processing happens through multiple DF threads. However, even if the number of threads per node is configured as 1, the same thread is assigned to partitions and can process in any order. This is intended by design.
Please note the processing of messages at a single partition level honors FIFO order,
Here are the alternatives:-
@LAHIRIG1 - FIFO processing in Queue Processors is not available OOTB and is being considered as an enhancement for future release. Detailed explanation from PEGA engineer -
Issue primary reason: Pega - Enhancement Issue primary reason description:
Pega OOTB is trying to add this sequential processing feature in the Infinity 24 version.
Explanation type: Answer/Information Provided Explanation description:
With the current design, multiple partitions are used for QueueProcessor, and Queue-For-Processing pushes the item to any partition and order is not guaranteed OOTB as QP design is intended for high throughput but not sequential.
Order of the processing is not guaranteed with a queue processor as the queued messages are pushed to any partition and parallel processing happens through multiple DF threads. However, even if the number of threads per node is configured as 1, the same thread is assigned to partitions and can process in any order. This is intended by design.
Please note the processing of messages at a single partition level honors FIFO order,
Here are the alternatives:-
1. Downsize the partitions count from 20/6 to 1, this way all the records are pushed to a single partition and processing happens in FIFO order. This will degrade the performance as QP works only using 1 thread 1 partition.
2. Pega OOTB is trying to add this sequential processing feature in the Infinity 24 version.
----------------------------
Additional option that I have tested and it works -
- Create a new topic in Kafka with multiple partitions - based on how many consumer threads are required.
- Map the topic to a queue class using Kafka Data Set. Add a partitioning queue used to sequence the messages like the case id. So all messages for the same case will go to the same partition.
- Instead of Queue-For-Processing, insert the record into Data Set, with Data Set-Execute.
- Create a Data Flow rule with source as Kafka Data Set and Destination as the Activity to be run in background (Queue Processor activity)
- Create a Real Time Data flow run to execute the data flow with as many threads as the partitions defined on the topic.
Please note - locking and primary page management, exception handling needs to be addressed within the activity unlike the Queue Processor which takes care of a lot of these aspects.
Tata Consultancy Services
AU
@BhanuPrakash_G - It is already Delayed. Issue is if the original requests come around the same time lets say with a few seconds difference (sometime milliseconds), the delayed items get scheduled around the same time as well. Furthermore if there is an outage all items are eligible to be picked up once the outage is over, irrespective of the original delay factor.
Accepted Solution
Tata Consultancy Services
AU
@LAHIRIG1 - FIFO processing in Queue Processors is not available OOTB and is being considered as an enhancement for future release. Detailed explanation from PEGA engineer -
Issue primary reason: Pega - Enhancement Issue primary reason description:
Pega OOTB is trying to add this sequential processing feature in the Infinity 24 version.
Explanation type: Answer/Information Provided Explanation description:
With the current design, multiple partitions are used for QueueProcessor, and Queue-For-Processing pushes the item to any partition and order is not guaranteed OOTB as QP design is intended for high throughput but not sequential.
Order of the processing is not guaranteed with a queue processor as the queued messages are pushed to any partition and parallel processing happens through multiple DF threads. However, even if the number of threads per node is configured as 1, the same thread is assigned to partitions and can process in any order. This is intended by design.
Please note the processing of messages at a single partition level honors FIFO order,
Here are the alternatives:-
@LAHIRIG1 - FIFO processing in Queue Processors is not available OOTB and is being considered as an enhancement for future release. Detailed explanation from PEGA engineer -
Issue primary reason: Pega - Enhancement Issue primary reason description:
Pega OOTB is trying to add this sequential processing feature in the Infinity 24 version.
Explanation type: Answer/Information Provided Explanation description:
With the current design, multiple partitions are used for QueueProcessor, and Queue-For-Processing pushes the item to any partition and order is not guaranteed OOTB as QP design is intended for high throughput but not sequential.
Order of the processing is not guaranteed with a queue processor as the queued messages are pushed to any partition and parallel processing happens through multiple DF threads. However, even if the number of threads per node is configured as 1, the same thread is assigned to partitions and can process in any order. This is intended by design.
Please note the processing of messages at a single partition level honors FIFO order,
Here are the alternatives:-
1. Downsize the partitions count from 20/6 to 1, this way all the records are pushed to a single partition and processing happens in FIFO order. This will degrade the performance as QP works only using 1 thread 1 partition.
2. Pega OOTB is trying to add this sequential processing feature in the Infinity 24 version.
----------------------------
Additional option that I have tested and it works -
- Create a new topic in Kafka with multiple partitions - based on how many consumer threads are required.
- Map the topic to a queue class using Kafka Data Set. Add a partitioning queue used to sequence the messages like the case id. So all messages for the same case will go to the same partition.
- Instead of Queue-For-Processing, insert the record into Data Set, with Data Set-Execute.
- Create a Data Flow rule with source as Kafka Data Set and Destination as the Activity to be run in background (Queue Processor activity)
- Create a Real Time Data flow run to execute the data flow with as many threads as the partitions defined on the topic.
Please note - locking and primary page management, exception handling needs to be addressed within the activity unlike the Queue Processor which takes care of a lot of these aspects.