Question
Cigna
IN
Last activity: 3 Jul 2024 9:58 EDT
How to create a file with multiple records using data flow?
We have a table with few thousand records in it, we want read each record from that table and format the data and move the formatted data to a file. All the rows from the table must be added to the single file.
Approach tried:
- Source of the data flow will be a database table data set
- Destination will be a file system
- Calling data transform in between to format the data
Issues faced:
- Its creating file for each record
Please help how to club all record together in a single file using data flows.
Accepted Solution
Updated: 3 Jul 2024 9:58 EDT
Pegasystems Inc.
NL
@RuchaS16869320this is because the partitioning key on the source data set will be used to partition the data and distribute the load across the data flow nodes to provide concurrent execution.
If you want files to only be written to a single node, you need to configure the source data set with no or a static partitioning key so there will only be a single partition. However, the way File data set works is that data is still flushed intermediately, and this would result in multiple files on the system
Eclatprime Digital Private Limited
AU
Can you please send me the configuration on how you are fetching and transforming the data. Ideally the data flow and the data set should be in the same class. When the data flow executes it will fetch single record and put it on the primary page and process that primary record. Basing on the mapping on the file data set 2nd tab it will create the file. So likewise it will process all the records at once and create single file only.
Infosys
IN
Hi @RuchaS16869320,
Use this article https://docs-previous.pega.com/decision-management/87/managing-data-flow-runs?
Regards,
Sai G
Cigna
IN
@RuchaS16869320 Updated the mapping tab, still its creating files on different nodes based on the thread. The requirement is to create a file on single node
Accepted Solution
Updated: 3 Jul 2024 9:58 EDT
Pegasystems Inc.
NL
@RuchaS16869320this is because the partitioning key on the source data set will be used to partition the data and distribute the load across the data flow nodes to provide concurrent execution.
If you want files to only be written to a single node, you need to configure the source data set with no or a static partitioning key so there will only be a single partition. However, the way File data set works is that data is still flushed intermediately, and this would result in multiple files on the system