We have millions of records in a table and we need to migrate those records using certain complex filtering to a new table. For the purpose of tracking and avoiding issues, we wanted to migrate a finite set of records daily for a certain period by calling the data flow in an activity and using that activity in a job scheduler.

The report configuration has all the required columns, filters, and max records (as per our use case) and a left join with the new class along with a filter on the new class so that subsequent runs only get us records that are not migrated to the new table.

The data flow configuration is simple, with the source as a report in the first class (all filtering done in the report only), then a convert shape to convert the class to the new class, and then the destination is a data set of the new class with config as insert only new records.

This approach overall is not working because whatever max records we are setting in the main report, the data flow while execution completely ignores the max records and processes the entire data in the first table. This defeats our use case.

While it considers all the columns and filters in the report, it only ignores the max records which is a bit strange.

Is there any way to restrict the data flow to execute only for a certain number of records at a time?