Question
Evoke Technologies
IN
Last activity: 4 Mar 2024 11:02 EST
Data Migration and Manipulation
We are working on an internal POC and looking for suggestions.
We need to migrate data from one of our internal Pega Application to another similar Pega environment. We were able to copy the data from one environment to another without any issues. The total amount of data is around 10 million.
Once the data is copied we need to manipulate few important attributes based upon business rules which is available outside of Pega application. For example if the Customer name is "John", then we pass this value to the external system which will convert "John" to "TestUser1" and this values needs to be saved back in Pega application.
Looking for suggestions on how to achieve this requirement effectively.
If we are creating activities/batch job then we need to loop through all 10 million records, call the business rules from the exposed services within the external system for data manipulation and then save to our Pega database which does not seems to be feasible option considering the amount of data.
Any suggestions or design approach will be really appreciated.
Pegasystems Inc.
GB
Given the large volume of data, it would be beneficial to use a batch processing approach. You could use a combination of Pega's Queue Processor and Data Flow features. Queue Processor can be used to process the data asynchronously in a distributed manner, which can significantly improve performance for large data sets. You can create a Queue Processor rule that fetches the data, calls the external system for data manipulation, and then saves the updated data back to the Pega database. Data Flow, a feature of Pega Customer Decision Hub, can also be used to process large volumes of data efficiently. It allows you to read, process, and write data in a distributed manner. You can create a Data Flow rule that reads the data from the database, calls the external system for data manipulation using a Data Transform or an Activity, and then writes the updated data back to the database. Remember to test and monitor the performance of your solution to ensure it meets your requirements.
⚠ This is a GenAI-powered tool. All generated answers require validation against the provided references.
Preparing for data migration > 2 Configure data migration settings
Given the large volume of data, it would be beneficial to use a batch processing approach. You could use a combination of Pega's Queue Processor and Data Flow features. Queue Processor can be used to process the data asynchronously in a distributed manner, which can significantly improve performance for large data sets. You can create a Queue Processor rule that fetches the data, calls the external system for data manipulation, and then saves the updated data back to the Pega database. Data Flow, a feature of Pega Customer Decision Hub, can also be used to process large volumes of data efficiently. It allows you to read, process, and write data in a distributed manner. You can create a Data Flow rule that reads the data from the database, calls the external system for data manipulation using a Data Transform or an Activity, and then writes the updated data back to the database. Remember to test and monitor the performance of your solution to ensure it meets your requirements.
⚠ This is a GenAI-powered tool. All generated answers require validation against the provided references.
Preparing for data migration > 2 Configure data migration settings
Best Approach to copy large amount of data from one table to other table.
Best way to copy 3 million records from external database to pega data table