Question
Bank Central Asia
ID
Last activity: 20 Dec 2024 16:34 EST
How to set the number of Partitions in Batch Outbound Scheduler Dataflow of Customer Decision Hub?
Hi,
We need to adjust the number of partitions of the batch outbound scheduler dataflow on Customer Decision Hub from the default 99.
We have 6 batch nodes with each 7 threads. On our daily scheduler, it will really slowing down when some of the nodes finished their partitions & only several with slow nodes working with the partitions left, which if it can be processed with the other idle nodes, it will be so much faster.
I have changed MKTSegPartitionCount & MKTEmailPartitionCount dynamic system settings & fully restart the server, but it still not working.
Can anyone tell me which configuration should i change for this need?
Thank you for your help.
Accepted Solution
Updated: 4 Jul 2024 2:39 EDT
Bank Central Asia
ID
Want to update about this question, I've got the solution from GCS. It can be addressed by updating the partitions count by using the formula.
The number of partitions = 3 * number of threads (Infrastructure > Services) * number of batch nodes.
SysAdmin > Dynamic System Settings > MKTSegPartitionCount has to be updated as per the above formula.
If the updated value is not taking effect, it could be due to the Refresh audience option before each campaign run is not configured. You can also Refresh the segment configured on the campaign manually and during the campaign run, verify if the partitions on the dataflow got updated. Try the above before increasing the number of threads or batch nodes.
Clarissa
Updated: 4 Jul 2024 2:39 EDT
Pegasystems Inc.
GB
To adjust the number of partitions for the batch outbound scheduler dataflow in Pega Customer Decision Hub, you need to create and configure a dynamic system setting. In the header, click Create > SysAdmin > Dynamic System Settings. Configure the parameters in the New tab. In the Owning Ruleset field, enter: Pega-Engine. In the Setting Purpose field, enter: prconfig/dsm/services/stream/pyTopicPartitionsCount/default. In the Value field, enter the new global default partitions count per topic. Save the changes and restart the server. Please note that the new setting is applied only to newly created partitions.
⚠ This is a GenAI-powered answer . All generated answers require validation against the provided references.
Changing the default number of partitions per topic
Managing Concurrent Campaign Data Flow Runs
==> @malaa1 @nairv1 please could you provide your input here?
Bank Central Asia
ID
@MarijeSchillern Hi Marije, thank you so much for your response. But there are another way by changing the MKTSegPartitionCount DSS & refresh the segment which are easier to implement in this case.
Accepted Solution
Updated: 4 Jul 2024 2:39 EDT
Bank Central Asia
ID
Want to update about this question, I've got the solution from GCS. It can be addressed by updating the partitions count by using the formula.
The number of partitions = 3 * number of threads (Infrastructure > Services) * number of batch nodes.
SysAdmin > Dynamic System Settings > MKTSegPartitionCount has to be updated as per the above formula.
If the updated value is not taking effect, it could be due to the Refresh audience option before each campaign run is not configured. You can also Refresh the segment configured on the campaign manually and during the campaign run, verify if the partitions on the dataflow got updated. Try the above before increasing the number of threads or batch nodes.
Clarissa
Updated: 20 Dec 2024 16:34 EST
Excellus BlueCross BlueShiled
US
@ClarissaL16661030 I understand that the factor of 3 is to distribute the load evenly .. but do we know why it is "3" . Shouldn't the no of partition be a multiple of (no of node * no of threads ). In your case - a multiple of 42 , like 84 ,126 ?