Thread leak in Elastic Search with Pega 7.3.1
There seems to be a thread leak in the version of Elastic Search used by Pega 7.3.1
I have a constant growth in live threads locked against an Elastic Search Reentrant Lock:
Stack Trace:
There seems to be a thread leak in the version of Elastic Search used by Pega 7.3.1
I have a constant growth in live threads locked against an Elastic Search Reentrant Lock:
Stack Trace:
Name: elasticsearch[73517F73CDA8220A61E5A3C9DDFACF59][generic][T#659] State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@7b7ac0f9 owned by: elasticsearch[73517F73CDA8220A61E5A3C9DDFACF59][[unicast_connect]][T#6] Total blocked: 0 Total waited: 3 Stack trace: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) org.elasticsearch.common.util.concurrent.KeyedLock.acquire(KeyedLock.java:70) org.elasticsearch.transport.netty.NettyTransport.disconnectFromNode(NettyTransport.java:1078) org.elasticsearch.transport.TransportService.disconnectFromNode(TransportService.java:274) org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$2$1.doRun(UnicastZenPing.java:258) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)
This is possibly a known issue, although I'm not sure that cluster members are joining / leaving that frequently:
https://github.com/elastic/elasticsearch/issues/8057
Accepted Solution
Pegasystems Inc.
US
Is this a multi-node environment? In earlier releases of Pega I have seen elasticsearch threads start accumulating when the nodes are unable to communicate with each other over the elasticsearch ports (9300-9400). This SA discusses a similar issue, but the resolution was to disable elasticsearch as the functionality wasn't needed: https://community.pega.com/support/support-articles/500-threads-spawned-pegarules-elastic-search
Link Group
AU
Here's a more accurate ES issue to track:
https://github.com/elastic/elasticsearch/issues/19370
... and a PR to include in Pega in the next release:
https://github.com/elastic/elasticsearch/pull/22277
Probably needs ES 5.x or 6.x
Accepted Solution
Pegasystems Inc.
US
Is this a multi-node environment? In earlier releases of Pega I have seen elasticsearch threads start accumulating when the nodes are unable to communicate with each other over the elasticsearch ports (9300-9400). This SA discusses a similar issue, but the resolution was to disable elasticsearch as the functionality wasn't needed: https://community.pega.com/support/support-articles/500-threads-spawned-pegarules-elastic-search
Link Group
AU
Thanks Nick... it looks like we missed out on opening up those TCP ports on the Windows Firewall on one of the Pega nodes, since it was added retrospectively (rather than through our build automation scripts). It seems to be much better now!