For a usecase we have to process millions of records when we go live. Once the whole data is processed, we would get delta of changes (from the upstream systems) which are very few in number and can be handled by the queue processors.
But the challenge i am facing is for the the first time processing. My current architecture comprises of couple of job schedulers queuing items to a Queue processor.
I have the queue processor setting to process items immediately and i have 10 threads to run per node. I have 3 stream nodes defined. (this is the configuration in the test environment)
It is mentioned that there cannot be more than 20 threads per cluster that can be run. Does production cloud servers in general have more than 1 cluster? Is there a way we can have more than 20 threads that can be run.
Also where can i check how many concurrent processes are being run?
Also, is there any way we can leverage the scaling of the servers (provided on the cloud) if the there is no way to increase the number of concurrent threads more than 20?