Problem on Kafka partitions associated to pyFTSIncremental indexer
Hello,
I am new to Kafka and i am trying why the Search does not work.
I have a VM with Pega 8.4. The problem is not on the Queue Processor pyFTSIncrementalIndexer is more on the data flow called pyFTSIncrementalIndexer that is accessible from the System > Search landing page and appears to be blocked and shows problems on Kafka partitions:
Opening the details i see the following exception:
Hello,
I am new to Kafka and i am trying why the Search does not work.
I have a VM with Pega 8.4. The problem is not on the Queue Processor pyFTSIncrementalIndexer is more on the data flow called pyFTSIncrementalIndexer that is accessible from the System > Search landing page and appears to be blocked and shows problems on Kafka partitions:
Opening the details i see the following exception:
Caused by: com.pega.fnx.stream.spi.StreamServiceException: Unknown stream PYFTSINCREMENTALINDEXER
at com.pega.fnx.stream.spi.impl.processor.StreamServiceExceptions.translateKafkaException(StreamServiceExceptions.java:56)
at com.pega.fnx.stream.spi.impl.processor.StreamServiceExceptions.translate(StreamServiceExceptions.java:40)
at com.pega.fnx.stream.spi.impl.processor.GetStreamPartitionsRequestProcessor.getDetails(GetStreamPartitionsRequestProcessor.java:98)
at com.pega.fnx.stream.spi.impl.processor.GetStreamPartitionsRequestProcessor.lambda$execute$0(GetStreamPartitionsRequestProcessor.java:56)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
When i starting Pega there are no issues. When instead i am looking on the PegaDATAFLOW.log and filter by pyFTSIncrementalIndexer i see a sequence of messages like these: first pyFTSIncrementalIndexer try to pick the partition after the partition is no more available.
2020-04-23 18:13:35,896 (.DataFlowDiagnosticsFileLogger) INFO - {"type":".RunStatusTransitionMessage","senderNodeId":"feeeef21337df7f3148f6c70310a8c96","timestamp":1587680015895,"messageId":"2f555b4e-a793-44ff-b78f-43c12d1ec2ef","runId":"pyFTSIncrementalIndexer","previousStatus":"PENDING_ASSIGNMENT","newStatus":"PENDING_PICKUP","originator":"AssignPartitionsTask","reason":"Assigned partitions for stream run"}
2020-04-23 18:13:59,906 (.DataFlowDiagnosticsFileLogger) INFO - {"type":".RunStatusTransitionMessage","senderNodeId":"feeeef21337df7f3148f6c70310a8c96","timestamp":1587680039904,"messageId":"bdcc3c59-c995-4700-bc02-ead9026326e5","runId":"pyFTSIncrementalIndexer","previousStatus":"PENDING_PICKUP","newStatus":"PENDING_ASSIGNMENT","originator":"CheckAssignmentsTask","reason":"No partition assignments found for run"}
2020-04-23 18:14:07,892 (.DataFlowDiagnosticsFileLogger) INFO - {"type":".RunStatusTransitionMessage","senderNodeId":"feeeef21337df7f3148f6c70310a8c96","timestamp":1587680047891,"messageId":"523f416e-0291-41db-acd6-0d947c06de4e","runId":"pyFTSIncrementalIndexer","previousStatus":"PENDING_ASSIGNMENT","newStatus":"PENDING_PICKUP","originator":"AssignPartitionsTask","reason":"Assigned partitions for stream run"}
I have tried to restart the Pega Server after decommission the Stream Node but nothing change. No errors appears on the logs except the following messages on repetition, like the QueueProcessor pyFTSIncrementalIndexer try to start but due of the problem on partitions it can't.
2020-04-23 17:20:28,120 [OBSCHEDULER_THREAD_4] [ STANDARD] [ ] [ ] ( QueueProcessorDataFlowHandler) ERROR - Queue processor: pyFTSIncrementalIndexer couldn't come to a stable state. Status: Initializing
2020-04-23 17:20:28,120 [OBSCHEDULER_THREAD_4] [ STANDARD] [ ] [ ] (ndexer.DistributedIndexerAgent) WARN - Started Queue processor: pyFTSIncrementalIndexer
2020-04-23 17:29:48,679 [ttp-nio-8080-exec-10] [dminStudio] [ ] [ GoGoRoad:01.01.01] ( QueueProcessorDataFlowHandler) ERROR localhost| Proprietary information hidden admin@gogoroad - Queue processor: pyFTSIncrementalIndexer couldn't come to a stable state. Status: Initializing
2020-04-23 17:31:13,081 [OBSCHEDULER_THREAD_4] [ STANDARD] [ ] [ ] ( QueueProcessorDataFlowHandler) ERROR - Queue processor: pyFTSIncrementalIndexer couldn't come to a stable state. Status: Initializing
2020-04-23 17:31:13,082 [OBSCHEDULER_THREAD_4] [ STANDARD] [ ] [ ] (ndexer.DistributedIndexerAgent) WARN - Started Queue processor: pyFTSIncrementalIndexer
How can i unblock the situation, and how to make the partition setup correct for pyFTSIncrementalIndexer ?
Thank you.
Regards,
Eliseo
***Edited by Moderator: Pallavi to change content type from Discussion to Question***
Updated: 24 Apr 2020 8:13 EDT
Pegasystems Inc.
US
Can you also attach the kafka server log? It should be under the log directory similar to this - zip everything if you can.
Can you also attach the kafka server log? It should be under the log directory similar to this - zip everything if you can.
root@pega-pega84mktaks-0:/usr/local/tomcat/kafka- Proprietary information hidden/logs# ls -l total 6976 -rw-r----- 1 root root 15018 Apr 23 00:02 controller-04-22-2020-1.log.gz -rw-r----- 1 root root 4709625 Apr 23 23:42 controller.log -rw-r----- 1 root root 0 Apr 22 20:22 kafka-authorizer.log -rw-r----- 1 root root 0 Apr 22 20:22 kafka-request.log -rw-r----- 1 root root 73390 Apr 23 23:45 kafkaServer-gc.log -rw-r----- 1 root root 102422 Apr 23 23:09 kafkaServer-gc.log.0 -rw-r----- 1 root root 102405 Apr 23 15:23 kafkaServer-gc.log.1 -rw-r----- 1 root root 102449 Apr 23 16:15 kafkaServer-gc.log.2 -rw-r----- 1 root root 102494 Apr 23 17:09 kafkaServer-gc.log.3 -rw-r----- 1 root root 102480 Apr 23 17:57 kafkaServer-gc.log.4 -rw-r----- 1 root root 102454 Apr 23 18:49 kafkaServer-gc.log.5 -rw-r----- 1 root root 102402 Apr 23 19:38 kafkaServer-gc.log.6 -rw-r----- 1 root root 102480 Apr 23 20:27 kafkaServer-gc.log.7 -rw-r----- 1 root root 102514 Apr 23 21:20 kafkaServer-gc.log.8 -rw-r----- 1 root root 102433 Apr 23 22:17 kafkaServer-gc.log.9 -rw-r----- 1 root root 172 Apr 22 20:22 log-cleaner.log -rw-r----- 1 root root 32123 Apr 23 00:02 server-04-22-2020-1.log.gz -rw-r----- 1 root root 104245 Apr 23 23:44 server.log -rw-r----- 1 root root 1079662 Apr 22 20:22 state-change.log
Updated: 24 Apr 2020 8:13 EDT
Pegasystems Inc.
IT
Hi Kevin,
In attachment you found the kafka logs in the zip and Pega logs.
Thank you.
Eliseo
Pegasystems Inc.
US
This is odd, while in pega logs complaining about
pySASIncrementalIndexer
There is nothing in the server.log, I also see old server.log*.gz
Question:
1. Did you delete the old kafkadata directory before this restart? (after truncating the relevant tables)
2. Do you observe the Partition directories under kafka related to PYSASINCREMENTALINDEXER?
Pegasystems Inc.
IT
Hello.
No, i did not delete the Kafka-data directory while Pega was before the restart.
I observe the partitions subdirectories under the Kafka-data dir for PYFTSINCREMENTALDIRECTORY, not see the ones for PYSASINCREMENTALINDEXER.
Anyway the log entry odd as you said is because i have tried to start the pySASIncementalIndexer queue processor.
In fact if i try to launch for example the Queue Processor pyFTSIncrementalIndexer from Admin Studio i have the message you see for pySASIncementalIndexer, this means:
which correspond the same entry in the Pega log.
2020-04-25 10:54:17,193 [OBSCHEDULER_THREAD_4] [ STANDARD] [ ] [ ] ( QueueProcessorDataFlowHandler) ERROR - Queue processor: pyFTSIncrementalIndexer couldn't come to a stable state. Status: Initializing
The partitions are there but seemed not associated to the data flow pyFTSIncrementalIndexer, that has still the same problem as reported.
Regards
Eliseo
HCL Technologies
IN
HI All,
Yes, I too face this issue ,so,I have deleted the old Koftha directory without deleting the tables .Could you please let me know what are the tables need to be deleted .
Thanks,
Bala
Elevance Health
US
Hi Team, we are also facing similar issue. Could you please let us know if there is any solution for the above problem ?
Regards,
Sravan
Cognizant Technology Solutions
IN
Did it resolved the issue. can you confirm?
Barclays
GB
Any luck on above issue. We are also facing same issue.
Kindly advise if this issue is resolved for you and share the steps you have followed for resolution.
LtiMindtree
PL
LtiMindtree
PL
Pega is not supporting embedded kafka actively, many unresolved kafka issues