Question
AI4Process
GB
Last activity: 19 Apr 2021 4:16 EDT
Push/Produce Data Stream to Pega Default kafka cluster throwing TimeoutException
I am trying use Kafka-Connect to push/produce stream data from twitter feed. I am trying to use the default Pega Kafka Zookeper and default pega kafka broker. I have created a new Topic in the default Zookeeper. But, While I am trying to produce the stream to Default kafka broker( to my newly created or any existing topic), I am facing time out exception -
D:\Pega 8.5.1\PRPCPersonalEdition\tomcat\kafka- Proprietary information hidden>bin\windows\kafka-console-producer --broker-list Proprietary information hidden:9092 --topic twitter_tweets --producer.config config\producer.properties --producer-property acks=all
ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2 >Hello Pega 13:16:11.004 [main] ERROR org.apache.kafka.clients.producer.internals.ErrorLoggingCallback - Error when sending message to topic twitter_tweets with key: null, value: 10 bytes with error: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. >^Z
Can anyone help me out here?
I do not want to use any external Kafka cluster for this purpose.
BNZ
NZ
@Chandramouli.M - If I understand your attempt correctly, you are trying to connect to KAFKA instance of twitter and then publish or consume messages from the topic on a twitter instance? Also you are using the Pega internal instances of broker/keeper. If so, can you let me know the below?
1. Your local broker instance will not have any connection established with the twitter topic?
2. What is the configuration you will have on your Kafka host setting?
3. what Kafka config you used on your Dataset instance ?
Please let me know.
Updated: 19 Apr 2021 4:16 EDT
AI4Process
GB
Here is what I am trying to achieve -
1. verified that my Pega 8.5 Instance is already running a default Kafka Cluster(Zookeeper) on port 2181.
2. verified that my Pega 8.5 Instance is already running a default Kafka Broker on port 9092
3. I have created a new topic "twitter_tweets" using following command from CLI -
kafka-topics --zookeeper Proprietary information hidden:2181 --topic twitter_tweets --create --partitions 3 --replication-factor 1
4. verified that the topic is created using following command -
kafka-topics --zookeeper Proprietary information hidden:2181 --list
5. tried to publish/produce message to the topic "twitter_tweets" using following command from CLI -
kafka-console-producer --broker-list Proprietary information hidden:9092 --topic twitter_tweets --producer.config config\producer.properties --producer-property acks=all
And received the below error message -
ERROR org.apache.kafka.clients.producer.internals.ErrorLoggingCallback - Error when sending message to topic twitter_tweets with key: null, value: 10 bytes with error: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
What I am trying to achieve next -
Here is what I am trying to achieve -
1. verified that my Pega 8.5 Instance is already running a default Kafka Cluster(Zookeeper) on port 2181.
2. verified that my Pega 8.5 Instance is already running a default Kafka Broker on port 9092
3. I have created a new topic "twitter_tweets" using following command from CLI -
kafka-topics --zookeeper Proprietary information hidden:2181 --topic twitter_tweets --create --partitions 3 --replication-factor 1
4. verified that the topic is created using following command -
kafka-topics --zookeeper Proprietary information hidden:2181 --list
5. tried to publish/produce message to the topic "twitter_tweets" using following command from CLI -
kafka-console-producer --broker-list Proprietary information hidden:9092 --topic twitter_tweets --producer.config config\producer.properties --producer-property acks=all
And received the below error message -
ERROR org.apache.kafka.clients.producer.internals.ErrorLoggingCallback - Error when sending message to topic twitter_tweets with key: null, value: 10 bytes with error: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
What I am trying to achieve next -
a) Use kafka-connect in stand alone mode to produce some data/stream to "twitter_tweets" topic. I have planned to use the Pega default Kafka cluster for this, it comes with "kafka-connect.sh" script.
b) consume data from "twitter_tweets" topic using Pega Dataset instance, if possible.
c) Use the Pega dataset as my source data in Pega case processing.
Here are the answers for your questions -
Q1. Your local broker instance will not have any connection established with the twitter topic?
Answer: yes, it looks like that.
Q2. What is the configuration you will have on your Kafka host setting?
Answer: No additional configuration is done for the Pega default Kafka cluster host.
I tried to create keystore and truststore files, but it did not work.
Q3. what Kafka config you used on your Dataset instance ?
Answer: No configuration is done yet.
Thanks for your response.
Best Regards