Tracing Techniques for the Pega Data Flows
Introduction
The attached document is the lesson learned for a future reuse with the Data Flow tracing techniques. It might be useful to see how to run the Pega Tracer to trace the execution of the Pega Data Flow run, or how to see the values populated on the records, while going thru the Data Flow run.
Data Flow
“Data flows allow you to sequence and combine data based on various sources and write the results to a destination. The sequence itself is established through a set of instructions and execution points from source to destination. Source and destination points can be abstract or driven by data sets and other decision data flows. Between source and destination, you can apply compose, convert, and merge strategy execution instructions.
The execution of data flows is done through work items which are instances of Pega-DM-DDF-Work and stored in the pc_work_dsm_batch table. The processing of data flows can be divided in two categories:
- Batch data flows using a database table as main input.
- Real-time data flows that, in active state, continue processing incoming stream data and requests made through the available service interface.
Data flow runs that are initiated through the Data Flows landing page run in the access group context. These data flows always use the checked-in instance of the Data Flow rule and the referenced rules. You can use a checked-out instance of the Data Flow if you initiate a local data flow run (by using the Run action in the Data Flow rule form) or a test run (a run initiated through the API).”
More about Pega Data Flows: https://community.pega.com/sites/default/files/help_v83/procomhelpmain.htm#rule-/rule-decision-/rule-decision-ddf/main.htm#Types_of_data_flows
TCS
IN
I am trying to find the use cases where this can be helpful.
1. Data migration from one system to another
2. ETL job in which Pega was not so efficient earlier
3. At times I have seen we need to do some validations, transformations based on user input, dependency checks while loading data in a data type. Can this be also useful for this use case?
4. Any other use cases?
Pegasystems Inc.
US
I attached the document, which you can review. Basically, when you are interested to:
- run Pega Tracer and see what exactly gets executed under the Pega Data flow, you need to know how to trigger the execution of the Pega Data Flow run in the user's requestor session
- see values populated on the records while going thru the Data Flow, you need to know how to stop the Data Flow with intentional failure in a correct place of the Data Flow
The use cases, which I attached helped our team to validate the executions of complex Data Flows (both batch and real-time) using loads with millions of the records on the input to those flows.
JP Morgan Chase
US
Hello,
I am exactly looking to trace the Data flow execution and for that calling the data flow with browse operation. I am getting the results in the results page but I dont see any of the Data flow execution steps in the tracer.
Any help would be greatly appreciated.
Basically I need to refer to the Run ID during the data flow execution and am not finding anything to get to that run ID during the data flow execution.
Pegasystems Inc.
US
You can create an Activity to run the Data Flow in the browse mode (not in start mode) and run it with tracer. The browse mode will execute the Data Flow using your requestor session, so you will see the execution of that Activity in the tracer. You can also run a tracer on a particular rule from the Data Flow, like Data Transform, so the tracer will not be very long.
The run ID can be either set manually to be a constant value like RUN-1, so every time, when the Data Flow run is executed, it reuses the same run ID, but having that, Pega overrides a history of that run, or the run ID can be set dynamically, so every execution of the Data Flow run gets a new run ID and Pega keeps all the Data Flow run history for analysis. When you execute the Data Flow run, either form the Data Flow run Landing Page, or from an Activity, or from a Data Flow using the Action-Run option, Pega creates an internal case for each execution of that Data Flow run and there are internal OOTB Pega classes, where you can find the run ID too, like Data-Decision-DDF-Progress, Data-Decision-DDF-RunOptions
Please check below:
Hope this helps.
Virtusa
US
Hi
How to access the Run ID inside a datatransform called from Data Flow. Which page does all DataFlow progress properties are stored when DataFlow is running.
Thanks
Pegasystems Inc.
US
This might be not so easy, but you can invoke the same flow with static run ID multiple times. Why you need to have dynamic run ID? Basically data flow run works with pyWorkPage, as it is an internal case to execute the flow. How you trigger that data flow run, is it triggered from activity?