Import large Data file(1 Million Records) to Pega Table
I have a CSV file with 1 million records, i need to import it to a data type in pega and this is not a one time activity. i need to repeat this on daily basis. we are currently processing records one by one through File Listener but that is taking hours to complete the job. is there any way to dump the whole csv at one as import wizard use to do but we need to do this through activity.
We are on cloud infrastructure and on 8.7.6 version.
Thanks in Advance.
UnitedHealth Group
US
@VishantC I had a similar usecase which was taking over 14hrs to finish. I was able to bring it down within 2.5hrs.
This is how it was done...
1. Split the file in 10 smaller files.
2. Use 10 threads of the file listener.
3. Push the data to a Queue Processor.
Another approach should work as well and give similar throughput if you can use a DataFlow to process the file. In this case, it will be better to split the file into 40 or more smaller parts (depending on the number of partitions an nodes). You will have to set up a Repository (You might already have one) where the files are dropped, and configure a DataSet to process the files. The DataSet will be the data source for the DataFlow.
If the question is how to split the file, it can be done by a serverless function or cron job.
Hope this will help.
Regards,
Pulak D.