Contributed by Paul Gentile
Symptoms
Scenario
Errors
Explanation
Environments
Solution
Related content
Symptoms
The scenario reported triggers the Errors.
Scenario
You have created a job scheduler called JS_BillIncrease that runs daily at 9:00 in the morning. The job scheduler activity runs the data flow by setting "Data-Decision-DDF-RunOptions" pyAppliesTo "Data class " and pyRuleName = "LoadBillIncrease". Then you call the Pega-provided activity pxRunDDFWithProgressPage so that the Pega engine will run the data flow LoadBillIncrease and add an entry to .
After you add the entry, the system loads data from the Pega staging table using Dataset. Then you filter out null records for primary keys, use Convert shape to map properties to the Pega spine table properties, and load data to the Pega spine using Dataset.
You have added the Primary key and foreign keys to the newly created Bill Increase table in the Oracle Pega Spine table. You check the Main Subscriptions table to see if Subscription_ID from Subscriptions Table has added the corresponding entry to the Bill Increase table in the spine. For a few records, the entry is not available in the Subscriptions table, and the Pega data flow throws the errors.
Errors
com.pega.dsm.dnode.impl.dataflow.resilience.policy.MaxFailedRecordsPerRunPolicy$ErrorThresholdReachedException: The run failed, because it exceeds the maximum number of failed record, which is currently set to 0
ORA-02291: integrity constraint (EXTERNALMKTDATA.COMMS_BILL_INCREASE_FK) violated - parent key not found
Explanation
You expect the Dynamic System Settings (DSSes) to control the runtime data flow.
The DSS Pega-DecisionEngine • dataflow/batch/failureThreshold was set to 1.
Updating the DSS Pega-DecisionEngine • dataflow/batch/failureThreshold value to a higher number, for example, 2000000000, skips records for which it does not find a corresponding entry and continues with other loads. When the data flow runs, it checks the maximum failure count until it reaches 2000000000, and the data flow does not fail. However, this approach does not work. Restarting the cluster also does not work.
Environments
The problem was reported for a Pega Platform™ version 8.8.2 on-premises environment.
Solution
When using pxRunDDFWithProgressPage to start a data flow during runtime, manually set Resilience - Failure Records in the data flow runtime page, or else it will default to 0.
pyResilience.pyFailPolicy.pyMaxAllowedFailRecords is the full string.
Related content
Pega Documentation – Decision Management and Customer Decision Hub
Processing data with data flows