Use case: We are using queue processors to subscribe for CTI events. Each message does a long poll REST call with a time out of 30 secs. We have 17 agents, default KAFKA partition is 20 and 21 threads(7 threads for 3 background nodes). So partition to thread is 1:1. If an event is received in the 30 seconds the thread continues processing the event else it timeouts and requeues for another 30 seconds. This continues till the agent logs out of CTI.

Issue faced:

When using default round robin or message key based queuing we see partition to message is not 1:1. 

Example: If operator 1 and Operator 2 are queuing messages and are both allocated partition 1, now when operator 1 is long polling 30 secs, operator 2 message in partition 1 is not processed till 1 receives and event or timeouts(introducing a lag). So when a new CTI event occurs(new call/agent state change), that event gets delayed.

Possible workaround:

What we are looking for is having an internal mapping table where each operator will be assigned a partition. When we queue a message, we want to pass partition id(instead of round robin allocation or message key allocation) so that each operator message to partition is 1:1. Apache KAFKA supports a producer to queue to a particular partition but somehow we do not see that option exposed in pega. 

Any insight on this will be appreciated?