Question
Mphasis
US
Last activity: 18 May 2017 17:46 EDT
Error 503: PegaRULES server not available
Hi,
We tried to recycle our JVMs and when we tried to check if the servers are up and running, we saw this error mentioned: "Error 503: PegaRULES server not available"
I checked SystemOut.LOG file and this is the error whenever I am trying to access the server:
[8/5/15 13:44:58:315 EDT] 000000ae webapp E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[WebStandard]: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: PegaRULES server not available
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:632)
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:662)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1301)
at com.pega.pegarules.web.impl.WebStandardImpl.sendOutput(WebStandardImpl.java:643)
at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:405)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
Hi,
We tried to recycle our JVMs and when we tried to check if the servers are up and running, we saw this error mentioned: "Error 503: PegaRULES server not available"
I checked SystemOut.LOG file and this is the error whenever I am trying to access the server:
[8/5/15 13:44:58:315 EDT] 000000ae webapp E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[WebStandard]: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: PegaRULES server not available
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:632)
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:662)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1301)
at com.pega.pegarules.web.impl.WebStandardImpl.sendOutput(WebStandardImpl.java:643)
at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:405)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:216)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:265)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:118)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:89)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3923)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:986)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1049)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:643)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
Accepted Solution
Mphasis
US
We deleted the empty 0 bytes PegaRULES-FirstLogCreationTime.log on all JVM’s and ALL JVM’s started successfully. Thanks to everyone else for helping as well.
Pegasystems Inc.
IN
Hi Arijit,
Can you delete all the prior historical Log files generated for Pega and Websphere for this current WAS profile and restart JVM?
Kindly keep backup of all these outside of Websphere directory.
Pegasystems Inc.
IN
Hi Arijit,
It would be good if you share complete Systemout and systemerr log.
Basic things to verify:
- Test connection is successful on PegRULES datasource.
- PRPC ear is shown as running in Application server.
Pegasystems Inc.
US
Right now, you have a diskspace issue. You need to clean that up and then restart the server. Post that log if the issue is still present.
Mphasis
US
Disk space was an issue earlier, but for eliminating that, we deleted the core dumps generated here: /websoft/IBMWebAS/profiles/profileServer/heapdumps/cl_app090_dev_xwasasdm1d (Except the one generated on today).
However after that when we restarted the JVMs, this issue started appearing.
Updated: 6 Aug 2015 6:47 EDT
Mphasis
US
It didn't help. This specific error is appearing in the systemout.log file that I want to highlight (New systemout.log file generated is attached along with):
[8/5/15 16:29:44:538 EDT] 0000005f SystemOut O 2015-08-05 16:29:44,538 [ server.startup : 0] [ ] [ ] [ ] ( context.web.WebEnvironment) INFO - Web tier using local JNDI context
factory = com.ibm.websphere.naming.WsnInitialContextFactory
url = corbaloc:rir:/NameServiceServerRoot
path = devasdcell/clusters/cl_app090_dev
[8/5/15 16:29:44:539 EDT] 0000005f PRBootstrap E com.pega.pegarules.internal.bootstrap.PRBootstrap Problem during method invocation (contextInitialized)
java.lang.StringIndexOutOfBoundsException
at java.lang.String.<init>(String.java:183)
at com.pega.pegarules.priv.util.LogDeleteUtil.readFromFile(LogDeleteUtil.java:296)
at com.pega.pegarules.priv.util.DailySizeRollingFileAppenderPega.handleFirstLogEvent(DailySizeRollingFileAppenderPega.java:167)
at com.pega.pegarules.priv.util.DailySizeRollingFileAppenderPega.subAppend(DailySizeRollingFileAppenderPega.java:148)
at com.pega.apache.log4j.WriterAppender.append(WriterAppender.java:159)
at com.pega.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
at com.pega.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at com.pega.apache.log4j.AsyncAppender.append(AsyncAppender.java:146)
It didn't help. This specific error is appearing in the systemout.log file that I want to highlight (New systemout.log file generated is attached along with):
[8/5/15 16:29:44:538 EDT] 0000005f SystemOut O 2015-08-05 16:29:44,538 [ server.startup : 0] [ ] [ ] [ ] ( context.web.WebEnvironment) INFO - Web tier using local JNDI context
factory = com.ibm.websphere.naming.WsnInitialContextFactory
url = corbaloc:rir:/NameServiceServerRoot
path = devasdcell/clusters/cl_app090_dev
[8/5/15 16:29:44:539 EDT] 0000005f PRBootstrap E com.pega.pegarules.internal.bootstrap.PRBootstrap Problem during method invocation (contextInitialized)
java.lang.StringIndexOutOfBoundsException
at java.lang.String.<init>(String.java:183)
at com.pega.pegarules.priv.util.LogDeleteUtil.readFromFile(LogDeleteUtil.java:296)
at com.pega.pegarules.priv.util.DailySizeRollingFileAppenderPega.handleFirstLogEvent(DailySizeRollingFileAppenderPega.java:167)
at com.pega.pegarules.priv.util.DailySizeRollingFileAppenderPega.subAppend(DailySizeRollingFileAppenderPega.java:148)
at com.pega.apache.log4j.WriterAppender.append(WriterAppender.java:159)
at com.pega.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
at com.pega.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at com.pega.apache.log4j.AsyncAppender.append(AsyncAppender.java:146)
at com.pega.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
at com.pega.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at com.pega.apache.log4j.Category.callAppenders(Category.java:206)
at com.pega.pegarules.priv.LogHelper.infoForced(LogHelper.java:1198)
at com.pega.pegarules.priv.context.web.WebEnvironment.getJNDIContexts(WebEnvironment.java:324)
at com.pega.pegarules.priv.context.web.WebEnvironment._runDiscovery_privact(WebEnvironment.java:260)
at com.pega.pegarules.priv.context.web.WebEnvironment.runDiscovery(WebEnvironment.java:166)
at com.pega.pegarules.web.servlet.WebAppLifeCycleListener._contextInitialized_privact(WebAppLifeCycleListener.java:269)
at com.pega.pegarules.web.servlet.WebAppLifeCycleListener.contextInitialized(WebAppLifeCycleListener.java:218)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:216)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:265)
at com.pega.pegarules.internal.web.servlet.WebAppLifeCycleListenerBoot.contextInitialized(WebAppLifeCycleListenerBoot.java:83)
at com.ibm.ws.webcontainer.webapp.WebApp.notifyServletContextCreated(WebApp.java:1704)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:411)
at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:169)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:749)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:634)
at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:426)
at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:718)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1177)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1370)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:968)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:776)
at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2195)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5477)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5603)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2200)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984)
at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
*** LOGS REMOVED 8/6/2015 ***
Hi Arijit,
Just before you ran out of disk space, I see the following error in the PegaRULES log:
2015-08-05 11:35:01,265 [.PRPCWorkManager : 4] [ STANDARD] [ ] [ PegaRULES:07.10] (nal.es.DistributedIndexerAgent) ERROR - Incremental indexing of 'DATA' entries failed: Indexing request failed. Please contact your system administrator.. Please contact your PegaRULES system administrator.
Do you know what was happening during this time?
Pegasystems Inc.
US
Are you using your own prlogging.xml? Is this intentional? I would suggest start up the engine with OOTB deployment (prlogging.xml would be inside prresources.jar) and go from there.
O Located PegaRULES logging configuration: file:/websoft/pega/config/prlogging.xml
Attach the file if that is intended.
Accepted Solution
Mphasis
US
We deleted the empty 0 bytes PegaRULES-FirstLogCreationTime.log on all JVM’s and ALL JVM’s started successfully. Thanks to everyone else for helping as well.
Pfizer
US
Hi,
Can you tell me where the PegaRULES-FirstLogCreationTime.log and PegaRULES-ALERT-FirstLogCreationTime.log files are present on the server.
We are not able to locate the files on the default log file location.