Request for Recommended Approach to Selectively Extract ~200M Records and Migrate Between Pega Clusters (Export to S3 → Import)
We are seeking guidance on the best-practice approach for extracting a large, selective subset of table data from one Pega cluster and importing it into another. Our requirement is as follows:
-
Data volume: Approximately 200 million rows across one or more Pega/Customer tables.
-
Goal:
-
Selectively extract table data based on specific business criteria (not a full database clone).
-
Package the extracted data in a format that can be exported to Amazon S3.
-
Re-import the data into the target Pega environment using a Pega-supported mechanism.
-
Considerations: These extract jobs will be running on Production instance.
HCA Healthcare
US
Use a Pega-supported extraction tool like BIX or a dedicated Data Flow that reads from the source tables with your business filters, so you only pull the 200M rows that match your criteria. Configure the extract so it writes to flat files (CSV/JSON/Parquet) on a mounted file system or S3-compatible storage, then have your infrastructure move or stream those files into Amazon S3. Run the extract in small batches (by date range, ID range, or partition key) and schedule it during off-peak hours to avoid overloading Production. On the target cluster, create matching table structures and use Pega’s supported import options (BIX import, Data Flows, or database bulk-load tools) to read the S3 files and write into the target tables. Validate row counts and spot-check sample records between source and target after each batch. Wrap the whole process with restartable jobs, logging, and monitoring so failed chunks can be retried without redoing everything.