Design Question for processing large data
This is more of a design question and your comments would be appreciated.
I am writing an Agent which needs to process about 300K users. For each of the users, my requirement is to enter a flow and create WorkObjects.
I have the users in a custom employee table in a different DB which I query to bring the user data back.
When I query this table, my query does a ' select * ' to select ALL the employees. I put this received data on the clipboard and the subsequent Agent Activities loop through each of the records (pxResults) on the clipboard to enter the flow and create the workobjects.
The question i have is it is possible to select about 3K records or less at once, process these employee records, select the next 3K records (so that the clipboard NOW has only these 3K employee records) process them and so on.
If possible to accomplish this HOW would I go about it. Any code examples would be extremely helpful.
Telstra
AU
Based on something, you can breakdown the data. For example, browse all users who start with letter "a" and then "b" all the way to "z".
Like this, choose other attributes which you can use to break down the data and your agent can go fetch those first and then remaining one by one.
Let me know if you have any questions. (send me a message)