Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 Miguel Angel Calderon Bravo (Miguel Calderon)
Plexus Tech
Senior System Architect (CSSA)
Plexus Tech
ES
Miguel Calderon Member since 2018 13 posts
Plexus Tech
Posted: Aug 28, 2019
Last activity: Aug 29, 2019
Posted: 28 Aug 2019 16:01 EDT
Last activity: 29 Aug 2019 7:07 EDT
Closed

Partition keys in Stream Data Set rules

Hi,
I created a Data Set to get messages from a kafka queue that has 2 partitions.
How do I assign value to the property specified as Partition Key?

Attached captures with the Data Set definition.

Thanks.

To see attachments, please log in.
Dev/Designer Studio
Data Integration

Posted: 6 years ago
Posted: 29 Aug 2019 5:39 EDT
Nishitha Chintala (chinn2)
PEGA
Principal Cloud Development Engineer, Management Plane Services
Pegasystems Inc.
IN

Hi Miguel Calderon,

What is the pega version that you are using?

From pega 7.4 ,by default number of partitions for kafka and stream dataset are 20(no configuration is needed). If you want to change this default number of partitions, go to kafka folder (in application server folder) and open connect-distributed.properties file and change the value of num.partitions.

By configuring partitioning you can ensure that related records are sent to the same partition. If no partition keys are set, the Kafka data set randomly assigns records to partitions.

Please go through below link for more information related to kafka data set creation.

Show More

Hi Miguel Calderon,

What is the pega version that you are using?

From pega 7.4 ,by default number of partitions for kafka and stream dataset are 20(no configuration is needed). If you want to change this default number of partitions, go to kafka folder (in application server folder) and open connect-distributed.properties file and change the value of num.partitions.

By configuring partitioning you can ensure that related records are sent to the same partition. If no partition keys are set, the Kafka data set randomly assigns records to partitions.

Please go through below link for more information related to kafka data set creation.

https://community.pega.com/sites/default/files/help_v74/procomhelpmain.htm

Generally to set the value of the property which is used as partition key for the dataset, you need to send the value in request itself.

Let's say i am using Age property as my partition key for stream data set, I need to send Age property value along with the required properties while sending the request to stream data set.

Hope it helps.Let me know if you have any queries. 


Show Less
To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice