Remove duplicates from a dataset
Hi All,
Is there any best way to remove duplicates from a dataset other than the below two. I found these two options,
Option 1: Create a dataflow and merge this with a report definition(without duplicates check) and write the results into destination dataset.
Option 2: Run a DB query on the dataset class using RDB. Run the query using scheduler and write the results into a dataset destination.
Any other quick and fast option for a dataset with huge number of records?
Pegasystems Inc.
GB
@Nizam One effective way to remove duplicates from a dataset is by adding a staging data set to the data job run.
This method allows you to perform record deduplication and other operations in the staging data set, which is faster and has a lower impact on performance because the volumes to process are smaller than in the destination data set. To remove duplicated records, you can select the "Insert new and overwrite existing records" checkbox in the Save options section of the staging data set configuration.