What is the Data Set timeout/number of records threshold?
We have a data set present in a data flow which extracts data from a data type, and sends it to an Azure repository. It happens every 24 hours via a job scheduler. The data type has millions of data. Lately we are seeing the extraction is failing, its going to the repair flow with the error "ProblemFlow_InfiniteInnerLoop". When we are trying to run it manually, it times out.
The questions we have are:
- What are max number of records that can be exported via a data set?
- What is the timeout threshold of a data set export?
Maantic Inc
IN
@SohamM95have you tried with a smaller number of set limit, does it work. If not see if there is any max insertion limitation on Azure repository that is causing problems, provide log snippet or go through it and do further analysis on the same
As far as i know the browse operation the same you can configure for Dataset , but for save or insertion there is no such max limit, try setting the limit on browse and insert the same see if it works.
Maantic Inc
IN
Smaller limit was working till now. The cause is that the number of records is crossing the limit. Its not a problem on the Azure repository side because the flow is not reaching that point.
When the data flow runs, it extracts the whole data set, I want to know what's the threshold value on the number of records and what is the timeout of the extraction.
Updated: 8 Apr 2024 1:51 EDT
Maantic Inc
IN
@SohamM95 I guess you should be getting pega Alert logs are you getting PEGA0007 ?? or any query timeout.
its on a prconfig.xml level and not Data flow level am afraid.
the default config is this 500ms per query have you guys updated this.
<env name="alerts/database/operationTimeThreshold" value="500" />.
This will throw alert log but threshold you have to see as in Pega doc also its not there about the max items/threshold as far as i remember that can be processed.
Check the query from pega first and try to run the same in DB oracle with query planner and see cost estimate there I guess you can backtrack the threshold and strain the system accordingly on the borderline.
Provide log snippet or else it will be tough to debunker this.
Maantic Inc
IN
We only got PEGA0118 AND PEGA0130 alerts. No exceptions found. We'll see after increasing the threshold of the DSSs mentioned in the documentation.