fix to clear the WebSphere tempWork directory?
We are using PEGA application in our company, but when ever this login in issue occers, we were told to delete the Maker file and shutdown all the JVMs in the cluster at the same time and start them one by one. Which is causing us the applicaiton down time and effecting business.
Is this the only way to clear the cache and delete the Marker file. If so this is a bug and which needs to be fixed.
If there is a fix for this please let me know.
Note: same resolution is posted in "https://pdn.pega.com/how-clear-websphere-tempwork-directory"
Thanks
Harsha
Prime Therapeutics
US
User not able to login, and we see that connection failed with the Data base in websphere logs.
So also checked with DB logs found nothing, checked with Webseal security server logs nothing found there.
But follow the procedure "https://pdn.pega.com/how-clear-websphere-tempwork-directory" including shutting down all JVMs same time, delete marker file and start the JVMs resolved the issue. but this issue is not one time thing, it happened more than few time in a week.
Prime Therapeutics
US
11/23/15 7:08:18:695 CST] 0000044a E UOW=null source=com.ibm.ws.webcontainer.webapp class="com".ibm.ws.webcontainer.webapp.WebApp method=logServletError org=null prod
=null component=null thread=[WebContainer : 1]
SRVE0293E: [Servlet Error]-[sso]: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: XXXXXX server not available.
Pegasystems Inc.
US
plz share the complete SystemOut/SystemErr files, this looks like a generic Websphere issue. Are you also using any SSO solution with Websphere?
Prime Therapeutics
US
Yes we are using SSO (webseal) for all application running in websphere.
But we checked logs in webseal it and it was not issue on webseal, because SSO was working all other application and no error in webseal.
But the issue was resolved only by the work around provided by PEGA, deleting marker file and clearing the cache and bouncing the JVMs.
What I am hopping is if you have come across similar issue in the past and able to have fix pack for it or an resolution.
Prime Therapeutics
US
[11/23/15 6:11:14:824 CST] 00001596 A UOW=null source=com.ibm.ws.security.registry.ldap.LdapRegistryImpl org=IBM prod=WebSphere component=Application Server thread=[Default : 182]
SECJ0419I: The user registry is currently connected to the LDAP server ldap://XXXXXXXXXXXXXXXXXXX:636.
[11/23/15 7:08:18:695 CST] 0000044a E UOW=null source=com.ibm.ws.webcontainer.webapp class="com".ibm.ws.webcontainer.webapp.WebApp method=logServletError org=null prod=null component=null thread=[WebContainer : 1]
SRVE0293E: [Servlet Error]-[sso]: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: PegaRULES server not available
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:643)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1189)
at com.pega.pegarules.web.impl.WebStandardImpl.sendOutput(WebStandardImpl.java:543)
at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:315)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
[11/23/15 6:11:14:824 CST] 00001596 A UOW=null source=com.ibm.ws.security.registry.ldap.LdapRegistryImpl org=IBM prod=WebSphere component=Application Server thread=[Default : 182]
SECJ0419I: The user registry is currently connected to the LDAP server ldap://XXXXXXXXXXXXXXXXXXX:636.
[11/23/15 7:08:18:695 CST] 0000044a E UOW=null source=com.ibm.ws.webcontainer.webapp class="com".ibm.ws.webcontainer.webapp.WebApp method=logServletError org=null prod=null component=null thread=[WebContainer : 1]
SRVE0293E: [Servlet Error]-[sso]: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: PegaRULES server not available
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:643)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1189)
at com.pega.pegarules.web.impl.WebStandardImpl.sendOutput(WebStandardImpl.java:543)
at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:315)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:216)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:265)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:118)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:89)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1694)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:970)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:508)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3994)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:945)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:516)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:307)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:278)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1049)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:643)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)
[11/23/15 7:13:38:726 CST] 00000449 E UOW=null source=com.ibm.ws.webcontainer.webapp class="com".ibm.ws.webcontainer.webapp.WebApp method=logServletError org=null prod=null component=null thread=[WebContainer : 0]
SRVE0293E: [Servlet Error]-[sso]: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: PegaRULES server not available
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:643)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1189)
at com.pega.pegarules.web.impl.WebStandardImpl.sendOutput(WebStandardImpl.java:543)
at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:315)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:216)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:265)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:118)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:89)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1694)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:970)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:508)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3994)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:945)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:516)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:307)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:278)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1049)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:643)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)
Pegasystems Inc.
US
Looks like you are having http 503 error: "PegaRULES server not available". I think these errors are just the victims when the Pega Etier/WebTier runtime environment is no longer usable (even if the process is still running). The clues should be before the so called login-errors happen. There are many reasons for this. So it is hard to say without further information. What if you restart the server without deleting those temp dirs? Was the recycling successful (i.e., Pega Engine starts up fine and you can login).
Prime Therapeutics
US
No if the JVMs are bounced with out deleting the marker files, JVMs comes up but application does not work as expected.
But if deleting marker files and cache from Data Base and bounce the JVMs it worked as expected.
Pegasystems Inc.
US
This is actually a good thing. if you can share the prpc startup log without deleting the marker file and cache, it should tell you what is wrong. Like I said earlier, you should find out why prpc engine does not start up correctly.