Slow Processing on Data Flow
Hi,
we are running data flow to processing account on start of day. we have 1 million data to process and user 14 nodes to parallel processing. we have done check the query is running well there are no heavy query time during the process. any suggest to tunning the process to become faster?
ai4process ltd
GB
To help identify the root cause of slow processing in your data flow, could you please provide more details on:
-
Which component or stage in the data flow is taking the most time? (e.g., source, transformation, destination)
-
Are there any custom functions or decisioning logic involved?
-
Is the slowness consistent across all nodes or isolated to a few?
-
What is the batch size and commit frequency configured?
-
Are you using external datasets like Kafka, or internal ones like Data Sets or DB tables?
This information will help pinpoint bottlenecks more accurately.
PT Bank Mandiri Tbk
ID
Hi @RaviChandra
-
Which component or stage in the data flow is taking the most time? (e.g., source, transformation, destination) > how to identify the component or stage that taking the most time?
-
Are there any custom functions or decisioning logic involved? > No, we are not used the custom functions or decission logic
-
Is the slowness consistent across all nodes or isolated to a few? > Is consistent in all nodes
-
What is the batch size and commit frequency configured? > we process more or less 45 data in one partition. where is configuration that I can find the commit freuency?
-
Are you using external datasets like Kafka, or internal ones like Data Sets or DB tables? > the source is from external data stagging. we use jdbc to inquiry the data.
Thank You
ai4process ltd
GB
You can find the component level statistics from the Data Flow run landing page. you can open via
Option 1:
- Configure --> Decisioning --> Decisions --> Data Flows
- You can find the Data flow instance under Batch/ Real-Time according to how you are running the Data flow
Option 2:
- If you are running it as a Batch Data flow and dont find the data flow instance under the Data Flows landing page, then Data flows can be found under instances of 'Pega-DM-DDF-Work' class.