Question
Acel Solutions
IN
Last activity: 10 Nov 2023 3:11 EST
How much records file listener can process?
Could you please let me know how much records file listener can process?
My Requirement: i have to read the .txt file in which 100K-200K records will be there and then we have to update into the table(Data-type) with some validation.
this process will be once in a month.
what will be the good way, please suggest. Thanks in advance
Infosys Limited
IN
I believe there is no such limiation for recording processing. It all depends on your processing design. In the file service processing activity create the page based on the received data and queue it to queueprocessror. The queueprocessor can take care of processing the multiple items at once.
Proximus
BE
Keep in mind to have a robust structure of your input file.
For such amount of data you should split reading the file and handling the data.
The point is that in case of bad data in any of the lines, processing of the file can break and it will move to .err and it can be a challenge to find out what the issue, what are the data to correct, which ones are OK and which are the ones to be corrected/reprocessed/resent. Then, next question for your process is how to identify and process the X% of failing data.
If the input file is perfectly structured and controlled, you'll then have less validation to do on your side.
Since it's a monthly process, I feel it must be first time right and perfectly controlled.
For the technical limitation, I guess it depends on what needs to be done with these data and the volume it represents.
Regards
Anthony