Queue-For-Processing - run in parallel
I'm using Queue-For-Processing to push a json asynchronously to AWS, can I make this run in parallel so when I get 1000's of incomming records processed without any delay & failure
Bentego
TR
Hi @KannanM3
First of all, where (external system DB?) and how (Exposed some service that is integrated by some external system to write some data (1000's record) to Pega DB table some like staging table then handled by another process?) are you getting 1000's incoming records?
Queue-For-Processing method has a "Queue Immediately" option like below. Maybe you can consider to check it and try again.
Regards.
Mert.
Idexx Laboratories
US
@m.caldag I have the Queue Immediately option checked as I'm using a dedicated Queue processor. The request comes through a Service REST in Pega which is called by another application.
Bentego
TR
@KannanM3that's fine then.
Probably, there might be some delays on processing time of the items due to many items in the queue even if queue immediately option of queue-for-processing method checked. Because, items are immediately queued and pending in scheduled table but processing time has some delays might be because of the many of the items in the queue. Can you consider to use some job (Job Scheduler rule) which is going to periodically work like hourly or daily or even minutes to call the connect rest rule to retrieve data and take the corresponding action through Job Scheduler activity? It means all the retrieved items will be processed within the same thread/execution of the activity without waiting processing time for each item like in queue processors.
Regards.
Mert.
Idexx Laboratories
US
@m.caldag Thanks Mert. I'll try out that, on a side note I see some of the json files generated as part of the DataSet that connects to the aws repository and drops the file in a aws bucket has multiple different records and sometimes empty file as well.
Is this happening because Pega runs the queue processor in parallel threads(configured to 5) per node and somehow it puts the record into one file and sends the other one empty?
Bentego
TR
Hi @KannanM3Could be.
Let's suppose we have four background nodes. If the same item queued for node 1 and then after some time later same item queued for node 2 which is going to process it. There will be duplicate work or some inconsistency depends on the action itself so it would be better to check scheduled/delay and broken tables if the item already queued or not. Job scheduler has a configuration to enable it with only one node for queue processors not sure that's why better to look those tables before adding them in the queue.
Regards.
Mert.