Reprocessing failed records in batch data flow runs
Hi All,
We need to rerun a data flow for failed records from batch processing. And we found in the article below that when you reprocess failures, you resubmit all the partitions that contain failed records to reprocess all the records that are on these partitions, whether the records failed or not during the run.
Is there any way to avoid processing success records from failed partitions and only rerun the failure records? Could someone please provide insights about this? Thank you.
Bits in Glass
IN
@AnggunMD In that article it says that to reprocess the failed records we need to resubmit the partitions with failed records, and it says it will process only the failed records and ignore the successful records. I am not much sure on that but I understood that way by going through the article.
Updated: 3 Mar 2023 3:09 EST
Anabatic Technologies Tbk Pt
ID
Hi, Thank you for your reply. Yes, but after we click Reprocess Failures, the Data Flow processes all the failure and successful records from the failed partition.
And in the note under step 7, it says, "When you reprocess failures, you resubmit all the partitions that contain failed records to reprocess all the records that are on this partition, whether the records failed or not during the run."
So, does the behavior when we click Reprocess Failures reprocess all failure and success records on each failed partition? Is there a way to reprocess the failed record without processing the successful record from the failed partition? Thank you