Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 Nadimuthu Thangavel (Nadimuthu T)
TATA Consultancy Services
Associate Consultant
TATA Consultancy Services
IN
Nadimuthu T Member since 2014 31 posts
TATA Consultancy Services
Posted: Dec 15, 2017
Last activity: Jan 30, 2018
Posted: 15 Dec 2017 1:10 EST
Last activity: 30 Jan 2018 7:42 EST
Closed
Solved

Thread Dump is happening due to java.util.concurrent.SynchronousQueue

HI Team,

I am getting below thread dump continously.

Please help me on this.

Show More

HI Team,

I am getting below thread dump continously.

Please help me on this.

2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - *** Thread id 990:WsSessionInvalidatorThread *** 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - sun.misc.Unsafe.park(Native Method) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.locks.LockSupport.park(LockSupport.java:198) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2055) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1097) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:819) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1080) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.lang.Thread.run(Thread.java:798) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - *** Thread id 78:Extension Profile Watcher *** 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - sun.misc.Unsafe.park(Native Method) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.locks.LockSupport.park(LockSupport.java:198) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2055) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:501) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:690) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - sun.nio.fs.AbstractWatchService.take(AbstractWatchService.java:131) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - com.wily.introscope.agent.extension.deployer.DeploymentWatcher.profileWatcherRun(DeploymentWatcher.java:96) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - com.wily.introscope.agent.extension.deployer.DeploymentWatcher.access$0(DeploymentWatcher.java:93) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - com.wily.introscope.agent.extension.deployer.DeploymentWatcher$1.run(DeploymentWatcher.java:86) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - java.lang.Thread.run(Thread.java:798) 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - 2017-12-15 11:52:40,922 [WebContainer : 37712] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) INFO - *** Thread id 626195:sonOutThreadPool : 96 ***


Show Less
To see attachments, please log in.
System Administration

Posted: 6 years ago
Posted: 15 Dec 2017 4:11 EST
Nadimuthu Thangavel (Nadimuthu T)
TATA Consultancy Services
Associate Consultant
TATA Consultancy Services
IN
Hi Team, Below mentioned exception occurred, after that thread dump happening continously. 2017-12-12 19:23:13,140 [WebContainer : 25544] [ ] [ ] [ ] ( fs.direct.FilesystemStorage) WARN - FileNotFoundException, calling reportProcessHandles java.io.FileNotFoundException:/TempDir/XXXXX/StaticContent/autogenerated/Rule-File-Bundle/desktopwrapper_13683327113.js_shrk (Too many open files) at java.io.FileInputStream.open(Native Method) ~[?:1.7.0] at java.io.FileInputStream.(FileInputStream.java:158) ~[?:1.7.0] at com.pega.pegarules.storage.fs.direct.FilesystemStorage._getInputStream_privact(FilesystemStorage.java:161) ~[prprivcommon.jar:?] at com.pega.pegarules.storage.fs.direct.FilesystemStorage.getInputStream(FilesystemStorage.java:136) ~[prprivcommon.jar:?] at com.pega.pegarules.storage.FileStorageManager.getInputStream(FileStorageManager.java:513) ~[prprivcommon.jar:?] at com.pega.pegarules.storage.FileStorageUtilsImpl.getInputStream(FileStorageUtilsImpl.java:120) ~[prprivate.jar:?] at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56) ~[?:1.7.0] at java.lang.reflect.Method.invoke(Method.java:620) ~[?:2.6 (10-05-2016)]
To see attachments, please log in.
Posted: 6 years ago
Posted: 15 Dec 2017 15:21 EST
Arun Kumar Mahanty (Arun_Mahanty)
Pegasystems Inc.
Principal Technical Solutions Engineer
Pegasystems Inc.
IN

Hello Muthu,

Could you please provide us the logs so that we can check what are the most threads which is locking the requestors .

There would be a culprit/root thread which is a root for a thread dump we need to identify that  .

Thanks,

Arun

To see attachments, please log in.
Posted: 6 years ago
Posted: 15 Dec 2017 6:31 EST
Ujjwal Kumar Shukla (UjjwalShukla_GCS)
PEGA
Senior Manager, Technical Support
Pegasystems Inc.
IN

Hi Muthu,
The best way to look at the issue would be to check the Thread that is runnable  and has acquired locks on synchronizers.

Thread dumps would give a snapshot of all the threads at that particular moment and we would have generally one thread in runnable that would be consuming all the resources.

However looking at your logs it seems that requested file is not accessible or remote server is very slow to respond back.

Thanks,
Ujjwal
 

To see attachments, please log in.
Posted: 6 years ago
Posted: 20 Dec 2017 10:02 EST
Nadimuthu Thangavel (Nadimuthu T)
TATA Consultancy Services
Associate Consultant
TATA Consultancy Services
IN

I have verified the file as mentioned the log file. Its available. 

"/TempDir/XXXXX/StaticContent/autogenerated/Rule-File-Bundle/"  This is the temp folder.

 

We are getting the issue at 2 nodes.

To see attachments, please log in.
Posted: 6 years ago
Posted: 20 Dec 2017 12:04 EST
Ujjwal Kumar Shukla (UjjwalShukla_GCS)
PEGA
Senior Manager, Technical Support
Pegasystems Inc.
IN

Hi Muthu,
Because we see below error

java.io.FileNotFoundException:/TempDir/XXXXX/StaticContent/autogenerated/Rule-File-Bundle/desktopwrapper_13683327113.js_shrk (Too many open files)
at java.io.FileInputStream.open(Native Method) ~[?:1.7.0]
at java.io.FileInputStream.(FileInputStream.java:158) ~[?:1.7.0]

Check your logic and make sure it is not opening too many files unnecessarily (for example, In a loop there is file open, but it is not getting closed anywhere).

Could you let us know the scenario that is causing this.

To understand it better we can refer to http://www-01.ibm.com/support/docview.wss?uid=swg21067352

Thanks,
Ujjwal

 

To see attachments, please log in.

Accepted Solution

Posted: 6 years ago
Posted: 23 Jan 2018 9:57 EST
Nadimuthu Thangavel (Nadimuthu T)
TATA Consultancy Services
Associate Consultant
TATA Consultancy Services
IN

Thanks Ujjwal,

We have increased the max file open count in operating system level. Now the issue resolved.

-Muthu

To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice