Question
Ford Motor Company
US
Posted: Mar 19, 2018
Last activity: Mar 20, 2018
Last activity: 20 Mar 2018 10:53 EDT
Closed
Solved
System Hung or Thread hung during Load Declarative Pega Definition Cache in WAS 8.5.5
Hi Team,
We have upgraded from 6.2 to 7.3, We are seeing this error and system is very slow.
Show More
Hi Team,
We have upgraded from 6.2 to 7.3, We are seeing this error and system is very slow.
[3/19/18 11:08:58:244 EDT] 00000089 SystemOut O 2018-03-19 11:08:58,243 [as3548.md1q.ford.com] [ STANDARD] [ ] [ ] ( external.async.IStartupTask) INFO - load Declarative Page Definition Cache ..starting [3/19/18 11:09:21:382 EDT] 00000082 ThreadMonitor W WSVR0605W: Thread "server.startup : 0" (00000066) has been active for 713448 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung. at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:719) at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1178) at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1383) at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639) at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:972) at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:777) at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2197) at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5486) at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5612) at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255) at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2202) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435) at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984) at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881) [3/19/18 11:09:21:383 EDT] 00000082 ThreadMonitor W WSVR0605W: Thread "server.startup : 1" (00000067) has been active for 713154 milliseconds and may be hung. There is/are 2 thread(s) in total in the server that may be hung. at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:719) at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1178) at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1383) at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639) at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:972) at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:777) at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2197) at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5486) at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5612) at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255) at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2202) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435) at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984) at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881) [3/19/18 11:09:21:385 EDT] 00000082 ThreadMonitor W WSVR0605W: Thread "eccvas3548.md1q.ford.com" (00000076) has been active for 713455 milliseconds and may be hung. There is/are 3 thread(s) in total in the server that may be hung. at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:127) at java.net.SocketInputStream.read(SocketInputStream.java:181) at java.net.SocketInputStream.read(SocketInputStream.java:152) at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1782) at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:4838) at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:6154) at com.microsoft.sqlserver.jdbc.SQLServerResultSet$CursorFetchCommand.doExecute(SQLServerResultSet.java:4903) at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:5696) at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1715) at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:180) at com.microsoft.sqlserver.jdbc.SQLServerResultSet.doServerFetch(SQLServerResultSet.java:4940) at com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(SQLServerResultSet.java:1058) at com.ibm.ws.rsadapter.jdbc.WSJdbcResultSet.next(WSJdbcResultSet.java:3114) at com.pega.pegarules.data.internal.store.rdbms.DatabaseResultSetForwardOnly.next(DatabaseResultSetForwardOnly.java:120) at com.pega.pegarules.data.internal.access.ResultSetIteratorImpl.<init>(ResultSetIteratorImpl.java:195) at com.pega.pegarules.data.internal.access.ResultSetPackagerImpl.packageResults(ResultSetPackagerImpl.java:206) at com.pega.pegarules.data.internal.access.Lister.listWithResultPackager(Lister.java:407) at com.pega.pegarules.data.internal.access.Lister.list(Lister.java:185) at com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:2941) at com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:2933) at com.pega.pegarules.exec.internal.declare.infengine.cbi.GlobalDeclarativeClass.getAllPropertyDefinitions(GlobalDeclarativeClass.java:359) at com.pega.pegarules.exec.internal.declare.infengine.cbi.GlobalDeclarativeClass.<init>(GlobalDeclarativeClass.java:244) at com.pega.pegarules.exec.internal.declare.infengine.cbi.GlobalCache.getGlobalClass(GlobalCache.java:704) at com.pega.pegarules.exec.internal.declare.infengine.cbi.DeclarativeCacheManagerImpl.getGlobalClass(DeclarativeCacheManagerImpl.java:1796) at com.pega.pegarules.exec.internal.declare.infengine.cbi.RuntimeDeclarativeClassCore.resolveRules(RuntimeDeclarativeClassCore.java:398) at com.pega.pegarules.exec.internal.declare.infengine.cbi.RuntimeDeclarativeClassCore.<init>(RuntimeDeclarativeClassCore.java:354) at com.pega.pegarules.exec.internal.declare.infengine.cbi.RuntimeDeclarativeClass.<init>(RuntimeDeclarativeClass.java:134) at com.pega.pegarules.exec.internal.declare.infengine.cbi.RuntimeCacheImpl.getRuntimeClass(RuntimeCacheImpl.java:313) at com.pega.pegarules.exec.internal.declare.infengine.cbi.DeclarativeCacheManagerImpl.getRuntimeClass(DeclarativeCacheManagerImpl.java:1638) at com.pega.pegarules.exec.internal.declare.infengine.cbi.ProcessingSupportImpl.getPropertyCandidates(ProcessingSupportImpl.java:108) at com.pega.pegarules.exec.internal.declare.infengine.cbi.ProcessingSupportImpl.getAutoChainCandidates(ProcessingSupportImpl.java:165) at com.pega.pegarules.data.internal.clipboard.DeclarativeSupportImpl.getAutoChainCandidates(DeclarativeSupportImpl.java:408) at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.autoChain(ClipboardPropertyImpl.java:1073) at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.autoChain(ClipboardPropertyImpl.java:951) at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyBase.autoChain(ClipboardPropertyBase.java:3423) at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.getValue(ClipboardPropertyImpl.java:4684) at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.getValue(ClipboardPropertyImpl.java:4675) at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyBase.getValue(ClipboardPropertyBase.java:3362) at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyImpl.toString(ClipboardPropertyImpl.java:5338) at com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDefinitionImpl.initialize(DeclarativePageDefinitionImpl.java:447) at com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDefinitionFactory.acquire(DeclarativePageDefinitionFactory.java:71) at com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDefinitionCacheImpl.initialize(DeclarativePageDefinitionCacheImpl.java:307) at com.pega.pegarules.session.internal.async.startup.DeclarativePageDefinitionsCacheTask.runTask(DeclarativePageDefinitionsCacheTask.java:21) at com.pega.pegarules.session.internal.async.startup.StartupTaskUtil.runTask(StartupTaskUtil.java:68) at com.pega.pegarules.session.internal.mgmt.PRNodeImpl.initializeSystem(PRNodeImpl.java:894) at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:379) at com.pega.pegarules.session.internal.PRSessionProviderImpl.getThreadAndInitialize(PRSessionProviderImpl.java:2084) at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineStartup.initEngine(EngineStartup.java:671) at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._initEngine_privact(EngineImpl.java:172) at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.doStartup(EngineImpl.java:145) at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.doStartup(EngineBean.java:121) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:507) at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370) at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411) at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224) at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:246) at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.doStartup(EngineBeanBoot.java:130) at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.doStartup(Unknown Source) at com.pega.pegarules.web.servlet.WebAppLifeCycleListener._contextInitialized_privact(WebAppLifeCycleListener.java:216) at com.pega.pegarules.web.servlet.AbstractLifeCycleListener._contextInitialized_privact(AbstractLifeCycleListener.java:159) at com.pega.pegarules.web.servlet.AbstractLifeCycleListener.contextInitialized(AbstractLifeCycleListener.java:73) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:507) at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370) at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411) at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224) at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:273) at com.pega.pegarules.internal.web.servlet.WebAppLifeCycleListenerBoot.contextInitialized(WebAppLifeCycleListenerBoot.java:83) at com.ibm.ws.webcontainer.webapp.WebApp.notifyServletContextCreated(WebApp.java:1709) at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:414) at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88) at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:170) at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:904) at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:789) at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:427) at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:719) at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1178) at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1383) at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639) at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:972) at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:777) at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2197) at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5486) at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5612) at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255) at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2202) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435) at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126) at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984) at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
***Edited by Moderator Marissa to update categories***
Show Less