Invalid configuration. Undefined stream provider end point.
Hello!
I have a Pega 8.4.1 application and it is throwing this exception:
(w.listener.RetryLoggerListener) ERROR - [pyBIBatchIndexClassesProcessor] Preparing retry 1091. Next attempt in 2m on data flow [pyBIBatchIndexClassesProcessor] on stage [InputStage] with affected partitions: [0, 1, 10, 11] java.util.concurrent.CompletionException: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point. at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:284) ~[?:1.8.0]
The stream node has a status of LEAVING.
The only way I know to fix this is to restart the application.
Is there anything I can do to prevent the system from getting into this state?
Thanks!
Fred.
Pegasystems Inc.
US
Good question but unless we know the root cause of your unavailable stream service, hard to answer. Can you share the kafka logs covering the incidence period, especially period before this error happens?
Capgemini
FR
Hi,
I have the exact same issue. After some days of running fine I get this exception, then my server is not responding properly, and it is restarting (hosted on kubernetes so restarting automaticaly with probes not repsonding)
Looking at the log just few seconds before for stream error I have this log :
@timestamp 2020-10-20T08:25:56.059Z @version 1 app level ERROR logger_name com.pega.dsm.dnode.api.server.StreamServerService message Unexpected broker state NORMAL, current env StreamServiceCluster[ MemberInfo{address=10.240.0.XXX, prpcId=pega-pegademo-xxxxxxxx-7vltj, state=NORMAL, sessionId=xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxd92eb4e}
] nodeType Stream,WebUser,BackgroundProcessing,Search,Batch,RealTime,Custom1,Custom2,Custom3,Custom4,Custom5,BIX,ADM,RTDG pegathread STANDARD source_host pega-pegademo-xxxxxxxxxx-7vltj stack tenantid thread_name PegaRULES-HeatlthMonitorTaskExecutor
(format is from json export of container logs)
then this exception, full content here for more details :
Hi,
I have the exact same issue. After some days of running fine I get this exception, then my server is not responding properly, and it is restarting (hosted on kubernetes so restarting automaticaly with probes not repsonding)
Looking at the log just few seconds before for stream error I have this log :
@timestamp 2020-10-20T08:25:56.059Z @version 1 app level ERROR logger_name com.pega.dsm.dnode.api.server.StreamServerService message Unexpected broker state NORMAL, current env StreamServiceCluster[ MemberInfo{address=10.240.0.XXX, prpcId=pega-pegademo-xxxxxxxx-7vltj, state=NORMAL, sessionId=xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxd92eb4e}
] nodeType Stream,WebUser,BackgroundProcessing,Search,Batch,RealTime,Custom1,Custom2,Custom3,Custom4,Custom5,BIX,ADM,RTDG pegathread STANDARD source_host pega-pegademo-xxxxxxxxxx-7vltj stack tenantid thread_name PegaRULES-HeatlthMonitorTaskExecutor
(format is from json export of container logs)
then this exception, full content here for more details :
@timestamp 2020-10-20T08:26:04.978Z @version 1 CorrelationId pyProcessCaseEvents app exception {"stacktrace":"java.util.concurrent.CompletionException: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point.\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)\n\tat java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)\n\tat java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)\n\tat java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)\n\tat com.pega.fnx.stream.spi.impl.metric.StreamSPIMetricsProxy.execute(StreamSPIMetricsProxy.java:41)\n\tat com.pega.fnx.stream.spi.impl.StreamSP.execute(StreamSP.java:68)\n\tat com.pega.fnx.stream.spi.loader.SPIProxy.execute(SPIProxy.java:48)\n\tat com.pega.platform.stream.StreamSPIStateValidator.execute(StreamSPIStateValidator.java:67)\n\tat com.pega.platform.stream.StreamSPIRequestValidator.execute(StreamSPIRequestValidator.java:57)\n\tat com.pega.fnx.stream.api.PollingSubscriber.poll(PollingSubscriber.java:80)\n\tat com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation.processRecords(StreamBrowseOperation.java:138)\n\tat com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation.access$000(StreamBrowseOperation.java:61)\n\tat com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation$1.emit(StreamBrowseOperation.java:107)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55)\n\tat com.pega.dsm.dnode.impl.dataflow.resilience.retry.RetryableBrowse$RetryableEmitter.produceRecords(RetryableBrowse.java:145)\n\tat com.pega.dsm.dnode.impl.dataflow.resilience.retry.RetryableBrowse$RetryableEmitter.emit(RetryableBrowse.java:111)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl$3.emit(DataObservableImpl.java:176)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55)\n\tat com.pega.dsm.dnode.api.dataflow.DataFlow$3.run(DataFlow.java:432)\n\tat com.pega.dsm.dnode.api.dataflow.DataFlow$3.run(DataFlow.java:425)\n\tat com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67)\n\tat com.pega.dsm.dnode.impl.dataflow.DataFlowThreadContext.lambda$submitInput$3(DataFlowThreadContext.java:228)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)\n\tat com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)\n\tat com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:59)\n\tat com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:56)\n\tat com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67)\n\tat com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:124)\nCaused by: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point.\n\tat com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getBootstrapServers(KafkaSettingsProvider.java:157)\n\tat com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.defaultConsumerProperties(KafkaSettingsProvider.java:80)\n\tat com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getConsumerProperties(KafkaSettingsProvider.java:62)\n\tat com.pega.fnx.stream.spi.impl.kafka.KafkaServerImpl.getConsumer(KafkaServerImpl.java:49)\n\tat com.pega.fnx.stream.spi.impl.processor.SubscriberRequestProcessor.getConsumer(SubscriberRequestProcessor.java:41)\n\tat com.pega.fnx.stream.spi.impl.processor.GetMessageRequestProcessor.execute(GetMessageRequestProcessor.java:75)\n\tat com.pega.fnx.stream.spi.impl.processor.GetMessageRequestProcessor.execute(GetMessageRequestProcessor.java:46)\n\tat com.pega.fnx.stream.spi.impl.StreamSPNative.execute(StreamSPNative.java:61)\n\t... 32 more\n","exception_class":"java.util.concurrent.CompletionException","exception_message":"com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point."} exception_class java.util.concurrent.CompletionException exception_message com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point. stacktrace java.util.concurrent.CompletionException: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point. at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870) at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251) at com.pega.fnx.stream.spi.impl.metric.StreamSPIMetricsProxy.execute(StreamSPIMetricsProxy.java:41) at com.pega.fnx.stream.spi.impl.StreamSP.execute(StreamSP.java:68) at com.pega.fnx.stream.spi.loader.SPIProxy.execute(SPIProxy.java:48) at com.pega.platform.stream.StreamSPIStateValidator.execute(StreamSPIStateValidator.java:67) at com.pega.platform.stream.StreamSPIRequestValidator.execute(StreamSPIRequestValidator.java:57) at com.pega.fnx.stream.api.PollingSubscriber.poll(PollingSubscriber.java:80) at com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation.processRecords(StreamBrowseOperation.java:138) at com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation.access$000(StreamBrowseOperation.java:61) at com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation$1.emit(StreamBrowseOperation.java:107) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) at com.pega.dsm.dnode.impl.dataflow.resilience.retry.RetryableBrowse$RetryableEmitter.produceRecords(RetryableBrowse.java:145) at com.pega.dsm.dnode.impl.dataflow.resilience.retry.RetryableBrowse$RetryableEmitter.emit(RetryableBrowse.java:111) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$3.emit(DataObservableImpl.java:176) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) at com.pega.dsm.dnode.api.dataflow.DataFlow$3.run(DataFlow.java:432) at com.pega.dsm.dnode.api.dataflow.DataFlow$3.run(DataFlow.java:425) at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) at com.pega.dsm.dnode.impl.dataflow.DataFlowThreadContext.lambda$submitInput$3(DataFlowThreadContext.java:228) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:59) at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:56) at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:124) Caused by: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point. at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getBootstrapServers(KafkaSettingsProvider.java:157) at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.defaultConsumerProperties(KafkaSettingsProvider.java:80) at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getConsumerProperties(KafkaSettingsProvider.java:62) at com.pega.fnx.stream.spi.impl.kafka.KafkaServerImpl.getConsumer(KafkaServerImpl.java:49) at com.pega.fnx.stream.spi.impl.processor.SubscriberRequestProcessor.getConsumer(SubscriberRequestProcessor.java:41) at com.pega.fnx.stream.spi.impl.processor.GetMessageRequestProcessor.execute(GetMessageRequestProcessor.java:75) at com.pega.fnx.stream.spi.impl.processor.GetMessageRequestProcessor.execute(GetMessageRequestProcessor.java:46) at com.pega.fnx.stream.spi.impl.StreamSPNative.execute(StreamSPNative.java:61) ... 32 more level ERROR logger_name com.pega.dsm.dnode.impl.dataflow.listener.RetryLoggerListener message [pyProcessCaseEvents] Preparing retry 1. Next attempt in 1s on data flow [pyProcessCaseEvents] on stage [InputStage] with affected partitions: [0, 1, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 2, 3, 4, 5, 6, 7, 8, 9] nodeType Stream,WebUser,BackgroundProcessing,Search,Batch,RealTime,Custom1,Custom2,Custom3,Custom4,Custom5,BIX,ADM,RTDG pegathread STANDARD source_host pega-pegademo-xxxxxxxxxxxx-7vltj stack tenantid thread_name DataFlow-Service-PickingupRun-pyProcessCaseEvents_Input:100, Access group: [PRPC:AsyncProcessor], Partitions=[0,1,10,11,12,13,14,15,16,17,18,19,2,3,4,5,6,7,8,9]
Capgemini
FR
Then I get an update on servers topology
Then I get an update on servers topology
Services topology has changed. Current topology: This node ==> ADM.Default pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 e8430c81-8f6f-4d2e-8370-d554da9c78a5 This node ==> Aggregation.Default pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 589ca581-f5d7-4ac3-b577-71e354dacfd2 This node ==> DataFlow.BackgroundProcessing pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 0a2e8192-5328-42c7-8041-7c84cd1911fd This node ==> DataFlow.Batch pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 1b5e520e-9645-43df-8d54-6e41285d070c This node ==> DataFlow.PegaMKTEmail pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 0cd435e7-c4d7-4b04-9bcd-b8d53b2f97ec This node ==> DataFlow.PegaMKTSMS pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 d8d16b84-3a15-4c3d-b987-9a19b95384d5 This node ==> DataFlow.RealTime pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 20498716-0723-4aea-bf21-d047eb61bb9a This node ==> DataFlow.Search pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 8047d3f8-be58-4a6f-b5cd-b1697b3b6743 This node ==> StreamServer.Default pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx STOPPED (was NORMAL) 8.4.2 87006bb9-4899-4b05-9399-72b1ad92eb4e This node ==> VBD.Default pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 fc334fcf-315b-42e0-8b0a-1e92b7e2deaf This node ==> pega pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx running 8.4.2 39486aed-64f8-4801-ab13-35d1f9f187c7
Capgemini
FR
can't get kafka logs as they are deleted when container is restarted.
Would be great if next Pega image could publish kafka logs as container logs !
Capgemini
FR
Well, got lucky and was able to download diagnostic laogs from Pega portal when I got the error logs again.See attached zip file generated by pega when the issue reproduced.
I am not a Kafka expert but not able to see anything wrong in those logs.
This time issue started on Pega logs at 2:27:19 PM
Capgemini
FR
Hi,
Created a support ticket for this issue INC-151877.
Fixed it by changing the helm char to use stateless instead of deployments, and applying a hotfix that fix an issue where long queries kill the process.
HFIX-65676
I still have the issue but much less.
Hope this help other people.
Regards,
Marc-Antoine
Pegasystems Inc.
GB
@FredG487 I also see this error in logs and I think that the root cause may be due to a Kafka failure. I see this error earlier in the logs:-
ERROR - Failed to start Kafka on 1 attempt, kafka log ERROR StatusLogger Unable to access file:/F:/Program%20Files/Apache%20Software%20Foundation/Tomcat%209.0/kafka- Proprietary information hidden/bin/file:F:/Program%20Files/Apache%20Software%20Foundation/Tomcat%209.0/kafka- Proprietary information hidden/config/log4j2.xml java.io.FileNotFoundException: F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka- Proprietary information hidden\bin\file:F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka- Proprietary information hidden\config\log4j2.xml (The filename, directory name, or volume label syntax is incorrect) at java.base/java.io.FileInputStream.open0(Native Method) at
......
@FredG487 I also see this error in logs and I think that the root cause may be due to a Kafka failure. I see this error earlier in the logs:-
ERROR - Failed to start Kafka on 1 attempt, kafka log ERROR StatusLogger Unable to access file:/F:/Program%20Files/Apache%20Software%20Foundation/Tomcat%209.0/kafka- Proprietary information hidden/bin/file:F:/Program%20Files/Apache%20Software%20Foundation/Tomcat%209.0/kafka- Proprietary information hidden/config/log4j2.xml java.io.FileNotFoundException: F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka- Proprietary information hidden\bin\file:F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka- Proprietary information hidden\config\log4j2.xml (The filename, directory name, or volume label syntax is incorrect) at java.base/java.io.FileInputStream.open0(Native Method) at
......
kafka.utils.Logging$class.$init$(Logging.scala:47) at kafka.Kafka$.<init>(Kafka.scala:30) at kafka.Kafka$.<clinit>(Kafka.scala) at kafka.Kafka.main(Kafka.scala)ERROR StatusLogger Reconfiguration failed: No configuration found for '2aae9190' at 'null' in 'null'12:00:18.879 [kafka-log-cleaner-thread-0] ERROR kafka.server.LogDirFailureChannel - Failed to clean up log for __consumer_offsets-0 in dir F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka-data due to IOExceptionjava.nio.file.FileSystemException: F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka-data\__consumer_offsets-0\00000000000000000000.log.cleaned -> F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka-data\__consumer_offsets-0\00000000000000000000.log.swap: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) ~[?:?]
......
Proprietary information hidden.jar:?] at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:291) [kafka_2.11- Proprietary information hidden.jar:?] at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) [kafka_2.11- Proprietary information hidden.jar:?]12:05:22.088 [LogDirFailureHandler] ERROR kafka.log.LogManager - Shutdown broker because all log dirs in F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka-data have failed
The only solution I have at the moment is to delete the Kaka folders under Tomcat and restart.