How does the system derive the number of partitions to execute a data flow? Say I have a node (10 cores) configured as a dataflow node. Say the thread count is set up to 1 when we create a Batch processing data flow work object. How many partitions would be the system create? Does that depend on the data? If so, how would the data affect the partitions created by the data flow?
Assume that we are processing 100 records with partition keys distributed across 0 to 9.
***Edited by Moderator: Pallavi to change content type from Discussion to Question***
Hey @mahar2, Thanks for the response. Could you explain the scenario that I mentioned in the actual post? The data set is associated with a database table. There are 100 records with equally distributed partition key ranging from 0 to 9. There is only one data flow node and the thread count on the data flow landing page is set to 1.
Posted: 3 years ago
Posted: 10 May 2020 11:12 EDT
Rakesh Mahapatra (mahar2)
Sr. System Architect