Default DataSet(Kafka)
Which dataset does pega use by default for stream(Asynchronous) processing? Let's say we want to setup a separate kafka cluster and leverage the same for stream processing(queue-for-processing method). For this scenario, which dataset will pega use behind the scenes for managing the processing of these queued records?
Updated: 6 Sep 2023 22:40 EDT
Ford Motor Company
US
To achieve high data consumption throughput and concurrency, the Stream service uses the Kafka concept of partitions. Starting in Pega Platform version 8.6.4, the default number of partitions per topic is six, which means that up to six clients can simultaneously consume data from a Stream data set.
Note: If the number of BackgroundProcessing or RealTime nodes is higher than six, then only six nodes are occupied. If more than one consumption thread per node is configured, then the number of consumers is the number of nodes multiplied by the number of threads.
To Change the default number of partitions per topic - Kafka
Pega-Engine • prconfig/dsm/services/stream/pytopicpartitionscount/default
To achieve high data consumption throughput and concurrency, the Stream service uses the Kafka concept of partitions. Starting in Pega Platform version 8.6.4, the default number of partitions per topic is six, which means that up to six clients can simultaneously consume data from a Stream data set.
Note: If the number of BackgroundProcessing or RealTime nodes is higher than six, then only six nodes are occupied. If more than one consumption thread per node is configured, then the number of consumers is the number of nodes multiplied by the number of threads.
To Change the default number of partitions per topic - Kafka
Pega-Engine • prconfig/dsm/services/stream/pytopicpartitionscount/default
RS:Pega-Engine
This setting will be used by Every New dedicated QP that will be created. new global default partitions count per topic.[A restart of the JVM is must in order to take DSS affective]