Queue processor - Failed to start
Queue Processor issue with external Kafka [KRaft mode]
I tried to configure Kafka with KRaft mode to see the behavior but ended up with Queue Processor . None of the QPs are able to start, getting error message "Queue processor, <<QP Name>> couldn't reach a stable state. Please check after some time."
It was working great with Kafka with Zookeeper. So I tried switching back to Zookeeper mode and also cleared the Kafka-data but still facing same error. Any solution on how to restore the QPs.
Thanks in advance.
Evonsys
IN
-
Check Configuration Settings: Ensure that all necessary configuration settings for Kraft mode are correctly applied. This includes settings for both controllers and brokers. Refer to the Confluent documentation for detailed configuration requirements
-
Review Logs: Examine the logs for any specific error messages or warnings that might give more insight into why the QPs are failing to start. Look for any discrepancies or missing configurations.
-
Verify Kafka Cluster Health: Ensure that your Kafka cluster is healthy, and all nodes are functioning correctly. In Kraft mode, it's crucial to have a stable controller quorum. Make sure you have an odd number of controllers (e.g., 3 or 5) to maintain quorum
-
Clear Kafka Data: Since you've already tried clearing Kafka data, ensure that all related directories and files are completely removed before restarting Kafka in Zookeeper mode. Sometimes residual data can cause issues.
-
Check Configuration Settings: Ensure that all necessary configuration settings for Kraft mode are correctly applied. This includes settings for both controllers and brokers. Refer to the Confluent documentation for detailed configuration requirements
-
Review Logs: Examine the logs for any specific error messages or warnings that might give more insight into why the QPs are failing to start. Look for any discrepancies or missing configurations.
-
Verify Kafka Cluster Health: Ensure that your Kafka cluster is healthy, and all nodes are functioning correctly. In Kraft mode, it's crucial to have a stable controller quorum. Make sure you have an odd number of controllers (e.g., 3 or 5) to maintain quorum
-
Clear Kafka Data: Since you've already tried clearing Kafka data, ensure that all related directories and files are completely removed before restarting Kafka in Zookeeper mode. Sometimes residual data can cause issues.
-
Check for Known Issues: There might be known issues or bugs related to running Kafka in KRaft mode. For instance, there have been reports of issues with Kafka in KRaft mode on certain platforms
2 [bitnami/kafka] Kafka with kraft mode not working · Issue #41393 · bitnami/containers · GitHub
. Checking the Kafka or Pega support forums might provide additional insights or solutions. -
Reconfigure Queue Processors: After switching back to Zookeeper mode, try reconfiguring your Queue Processors from scratch. This might help in resetting any misconfigurations that occurred during the switch.
HCA Healthcare
US
@SupriyaSK For QP not starting after switching Kafka to KRaft mode and back to Zookeeper mode, follow these steps to restore them. First ensure Kafka and Zookeeper are running correctly and their configurations are accurate. Clear Kafka logs and Zookeeper data using commands like rm -rf /path/to/kafka-logs/* and restart both services. Verify the Pega Kafka configurations in Data-Admin-Kafka instances and revalidate the Queue Processor rules. Clear Pega rule caches from the Cluster Management console. Check the pr_sys_statusnodes table for stale node entries and remove them if necessary. Ensure required Kafka topics exist using kafka-topics.sh and create any missing ones. Enable DEBUG logs for Queue Processors in Pega to gather detailed error information. Test connectivity between Pega nodes and Kafka to confirm they can communicate. If the issue persists, perform a full clean-up by stopping all services, clearing logs and data, and restarting Zookeeper, Kafka, and Pega nodes in sequence. Finally, verify the Queue Processor status in Admin Studio and restart them if needed. You can also open a Pega SR if non of them works