Pega 8.6.2 Tomcat on Oracle fails to shutdown properly
Hi All,
When we come to shutdown tomcat on production we get the PegaRules.log message "PegaRULES shutdown complete" but tomcat doesn't shutdown.
Instead what we see is that catalina won't quit and we see the following in the catalina log.
Hi All,
When we come to shutdown tomcat on production we get the PegaRules.log message "PegaRULES shutdown complete" but tomcat doesn't shutdown.
Instead what we see is that catalina won't quit and we see the following in the catalina log.
27-Sep-2022 23:07:44.364 INFO [node] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
27-Sep-2022 23:07:44.412 INFO [node] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
27-Sep-2022 23:07:44.413 INFO [node] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-jsse-nio-8443"]
27-Sep-2022 23:07:44.413 INFO [node] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-jsse-nio-8443"]
Exception in thread "elasticsearch[node][generic][T#5]" java.lang.NoClassDefFoundError: org/elasticsearch/transport/TransportService$TimeoutInfoHolder
Exception in thread "elasticsearch[node][generic][T#1]" at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:954)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.transport.TransportService$TimeoutInfoHolder
at com.pega.pegarules.bootstrap.loader.PRAppLoader.loadClass(PRAppLoader.java:598)
at com.pega.pegarules.bootstrap.loader.PRAppLoader.loadClass(PRAppLoader.java:550)
... 5 more
java.lang.NoClassDefFoundError: org/elasticsearch/transport/TransportService$TimeoutInfoHolder
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:954)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "elasticsearch[node][generic][T#6]" java.lang.NoClassDefFoundError: org/elasticsearch/transport/TransportService$TimeoutInfoHolder
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:954)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "elasticsearch[node][generic][T#3]" java.lang.NoClassDefFoundError: org/elasticsearch/transport/TransportService$TimeoutInfoHolder
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:954)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "elasticsearch[node][generic][T#7]" java.lang.NoClassDefFoundError: org/elasticsearch/transport/TransportService$TimeoutInfoHolder
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:954)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "elasticsearch[node][generic][T#8]" java.lang.NoClassDefFoundError: org/elasticsearch/transport/TransportService$TimeoutInfoHolder
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:954)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Based on this we can see that the connectors have shutdown but still tomcat refused to shutdown. We then see the following logs repeat over and over again.
27-Sep-2022 23:08:37.081 WARNING [m3-json-metrics-publisher] io.micrometer.core.instrument.push.PushMeterRegistry.publishSafely Unexpected exception thrown while publishing metrics for JsonMeterRegistry
java.lang.NullPointerException
at org.apache.logging.log4j.core.Logger.isEnabled(Logger.java:170)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1982)
at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:750)
at com.pega.gcs.m3.metrics.JsonMeterRegistry.publish(JsonMeterRegistry.java:216)
at io.micrometer.core.instrument.push.PushMeterRegistry.publishSafely(PushMeterRegistry.java:52)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
In the end after waiting about 20mins we have to manually kill tomcat with kill <PID> -9 isn't needed.
This is obviously slowing down our maintenance window as it happens on nearly all the non webuser nodes.
Anyone got any idea how to address this issue?
Thanks Craig
***Edited by Moderator Marije to add Support Case Details for 3rdparty metrics error***
***Edited by Moderator Marissa to update Support Case details***
Updated: 28 Sep 2022 16:20 EDT
Pegasystems Inc.
US
Not clear as to why elasticsearch appears to be preventing clean shut downs for tomcat.
The attached script has been used with CentOS/RHEL systems for consistent shut downs
update: the system won't allow me to paste the script text here, or attach the script even with a .txt extension.
This article explains how to do this for nginx, php, and mysql. You would grep '[o]pt/tomcat/bin' instead, otherwise the example commands should work fine
Vastcom
CN
I got this issue as well, I mean the m3-json-metrics-publisher warning. Then need to manually kill.
Do you resolve it?
I asked in GCS portal, the support replied that is related to third-party software. But I checked with my teammates, they said there is no micrometer is running in the RedHat system. I doubt that may from the Tomcat.
Virgin Media
GB
No still not found a solution to this issue. Happens even on the latest tomcat 9.0.74 which I've just installed.
Strange thing is it doesn't happen on all our systems, just some of them which is very odd.
LTIMindtree
SA
Issue looks new to me. Did you have access to Tomcat Ui Directly? Did you try to Kill the PID's from it ?
Virgin Media
GB
@Kishore Sanagapalli Sorry no, we shut down tomcat via command line using the shutdown.sh script.
My assumption at this point is that during a shutdown Pega is not shutting down the stream node because I can still see the open ports for elasticsearch attached to tomcat PID and yet the tomcat http/https ports are shutdown as confirmed with the catalina logs.
LTIMindtree
SA
Your case and assumption is possible if you're using external elastic search in your environment. May be you should report that to Pega through an INC for permanent Solution.