Using queryable Data Pages in activity
Hi,
In my use case I'm using a functionality similar to exporting a result set to CSV (think of the OOTB activity pxConvertResultsToCSV which you need to provide a pxResults). However, due to large result sets its giving performance issues.
Our client suggested sending the CSV files in chunks, e.g., result_2023-06-28_01.csv, result_2023-06-28_02.csv, ..., result_2023-06-28_04.csv. Where each file should have for example 250,000 lines for a total of 1,000,000 records.
To achieve that, I was thinking on using queryable data pages (that were introduced in recent versions of Pega 8.x), however, I think it can only be used in tables referenced in UI views. So, my question is, can this functionality be leveraged in activities? I wish that just like table pagination (in coordination with queryable data pages) fetches only the amount of records needed, I can split the pxResults returned by my Data page in chunks and then generate the files I need.
Any thoughts or ideas are welcome.
Thanks,
-Jose Navas
Bits in Glass
IN
@josenavas can you try to do your file generation using queue processor? Also is the source of data -> data type?
SoftServe
US
Hi @Anoop Krishna,
I'm using a Job scheduler set to run in a background node, but even there the result set is too large. About the source of the data, it's Work instances.
Thanks