Dataset execute kafka in produce mode while server connexion off
Could you please help in answering the following questions?
What happen when I wrote a kafka message with datatset execute while the connexion is broken ?
->When I tried the activity was a success and there is no details in the tracer on the dataset method
-> I guess there is no testing on the kafka connection
->but the issue is that there is no messages written in the kafka topic when I reconnect the node / kafka.
-> all the messages send during the non connexion period are lost
The client requires that all the messages are sends or stock for replay. How could I handle this kind of error ?
Stéphane
article related and non answered :
***Edited by Moderator Marije to add Capability tags***
Bits In Glass
GB
@gaillards Can you write activity with Dataset-Execute and see if any errors are found in logs.
configure a real-time stream and provide the kafka dataset, by default it has 3 retries if there is any issue such as connection/Time out it auto retries.
check the dataflow for any errors.
Also check https://docs-previous.pega.com/decision-management-reference-materials/troubleshooting-kafka
Updated: 14 Jun 2022 9:09 EDT
Macif
FR
@Sesha_Kandurihello, thank you for your suggestion, I will check a solution using a stream.
Concerning my main issue, I manage to get an error on the dataset execute :
-> there is a DSS to set in the Pega-DecisionEngine RS, dataset/execute/save/statusFailOnError to set to true
update :
We finally adress this need with a queue processor. We overload the pega standard queue to get our own replay timing and number of replay.
So the kafka dataset-execute is directly put in the activity of the queue :
-> if kafka recover within the replay time, the message will be send
-> if the queue reach the max replay it will fall in the broken process and need to be replay manually.