Question
Coforge
US
Last activity: 27 Apr 2021 13:24 EDT
Dataflow background processing read retries repeatedly and never succeeds with no exceptions
We are running a dataflow in background processing mode. It's successful for all the partitions but a few get stuck up in retry. The thing is connectivity to the dataset is quite good and bandwidth is also good. But only for a few segments, it keeps on trying again in the even lifecycle it shows it as successful (browse retry successful) but never completes (tries again) and this just goes on in a loop until I stop the process manually.
The irony is the run does not throw any exceptions in the logs or there are no failed records as well. Has anyone encountered issues like this? Any suggestion would be highly appreciated.
Messages in the life cycle report:
1. "Could not perform database dataset operation"
2. "Browse retry successful"
The above two just keep going on throughout for what I can say are hung partitions.
Accepted Solution
Updated: 27 Apr 2021 13:24 EDT
Coforge
US
We discovered that this issue was happening when the records to be processed under a partition were too high completed to the batch size configured for data flow processing.
Partition Record Count >> Pega batch size (configured in DSS).
So we reduced the number of records to be 8 to 10 times that of thread size and this issue was solved.
Coforge
US
Yes, there are no errors in the work object. Failure records # is 0.
But in the event report, we can see that the dataflow keeps on trying for a few partitions. We are running those partitions alone to debug the issue, but still, there are no errors all it says is
1. "Could not perform database dataset operation"
2. "Browse retry successful"
And it just runs without failing.
Coforge
US
One more thing to add. This is only happening for partitions where the partition size = 10 times the batch size in Pega (structureddata/defaultBatchSize).
We don't have any handles to reduce the size, but we are figuring out a way to reduce the size.
Accepted Solution
Updated: 27 Apr 2021 13:24 EDT
Coforge
US
We discovered that this issue was happening when the records to be processed under a partition were too high completed to the batch size configured for data flow processing.
Partition Record Count >> Pega batch size (configured in DSS).
So we reduced the number of records to be 8 to 10 times that of thread size and this issue was solved.