Data Flows- Partitioning Key
We have a huge data loaded into DDS table and from that source another DF is reading it. We have configured 20 Threads for the Data flow and have 2 Batch Nodes with created 80 Partitions. But issue is when the DF is processing its processing all the records in single partition which is taking time to process all the records.
Can some one please help on how to distribute the data across all the partitions during the DF processing when the source is DDS. If the source is Database Table mapped dataset then we can configure the partition key but not having that option for the DDS.
JP Morgan Chase
US
Can some one please respond on how to enable partitioning when the DDS is the source of the Data flow. We have 100K Plus records in the DDS but all the records are processing in just one partition.
When the Source is Database Table Mapped DS there in the Dataset we can specify the partitioning key but not having that option in the DDS. Any response would be much appreciated.
DCS
GB
Hi @satsihc6,
There is no way to add partition unlike DB Datset or Report definition. We got similar issues in the past, we were able to address the issue.
The first key defined in the DDS key list is used to create partitions and to distribute data across multiple decision data nodes. Please check the very first key defined in your DDS. It shouldn't be a constant value. If it's, make sure you are re-ordering the keys so that the very first key had a variance to build the partitions & distribute the load across batch nodes in the cluster. Hope this helps!
Thanks,
Nanjundan Chinnasamy
Pega Lead Decisioning Architect | 1:1 Customer Engagement | DCS
Updated: 28 Jun 2023 6:03 EDT
Maantic Inc
US
@satsihch Pega documentation shows, we can create key DDS dataset.
could you check below link
https://docs-previous.pega.com/decision-management/87/creating-decision-data-store-data-set