Question
Transport for NSW
AU
Last activity: 29 Oct 2020 12:48 EDT
Tomcat is not up after upgrading to Pega 8.4.1 from Pega 8.32 - problem with prweb.war deployment
Hi,
We have done an in-place upgrade from Pega 8.32 to Pega 8.41 successfully. However, after the upgrade when we started the app server (tomcat), there are problems with prweb.war deployment and we're unable to access pega application post upgrade.
Before staring the tomcat, we replaced the prweb.war and prhelp.war files from pega_841 build.
Our DB is Oracle 12C and so used the compatible ojdbc7.jar. Even tried with ojdbc8.jar but no luck.
Below is the log snippet showing errors. Full log is also attached.
I've tried connecting to DB using sqlplus manually and I didn't see any problems connecting to database using the same jdbc driver. Was able to query the table PEGA_RULES_83.pr_engineclasses using sqlplus successfully - queried this as log points that it is not able load a class from this table. I queried tables in data schema and rules schema without any issues.
Though our schemas names show pega_rules_83, it is upgraded to 84 - so don't be confused.
There are no privleges issues as well with db users because with the same privileges it was working fine with pega 8.3.2
Though log seems to be pointing to DB related configuration, I am not sure if that is the issue because the same configuration was working fine with pega 8.32 before the upgrade.
Though it looks more like some configuration issue, I couldn't find the root cause.
Any quick help is appreciated.
=====================================
Hi,
We have done an in-place upgrade from Pega 8.32 to Pega 8.41 successfully. However, after the upgrade when we started the app server (tomcat), there are problems with prweb.war deployment and we're unable to access pega application post upgrade.
Before staring the tomcat, we replaced the prweb.war and prhelp.war files from pega_841 build.
Our DB is Oracle 12C and so used the compatible ojdbc7.jar. Even tried with ojdbc8.jar but no luck.
Below is the log snippet showing errors. Full log is also attached.
I've tried connecting to DB using sqlplus manually and I didn't see any problems connecting to database using the same jdbc driver. Was able to query the table PEGA_RULES_83.pr_engineclasses using sqlplus successfully - queried this as log points that it is not able load a class from this table. I queried tables in data schema and rules schema without any issues.
Though our schemas names show pega_rules_83, it is upgraded to 84 - so don't be confused.
There are no privleges issues as well with db users because with the same privileges it was working fine with pega 8.3.2
Though log seems to be pointing to DB related configuration, I am not sure if that is the issue because the same configuration was working fine with pega 8.32 before the upgrade.
Though it looks more like some configuration issue, I couldn't find the root cause.
Any quick help is appreciated.
=====================================
28-Apr-2020 18:19:04.154 INFO [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRMiniLoader. Will load phase 2 bootstrap from Pega-EngineCode:08-04-02 28-Apr-2020 18:19:04.200 WARNING [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRMiniLoader. com/pega/pegarules/internal/bootstrap/phase2/PRBootstrapImpl.class could not be loaded from PEGA_RULES_83.pr_engineclasses for '08-04-02' codeset. 28-Apr-2020 18:19:04.200 INFO [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRMiniLoader. Could not find file:/app/tomcat/tomcat4poc/webapps/prweb/WEB-INF/lib/prbootstrap-8.4.1-359.jar 28-Apr-2020 18:19:04.201 INFO [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRMiniLoader. Perhaps executing with a non-standard classpath (e.g. JUnit tests)? 28-Apr-2020 18:19:04.210 INFO [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrap. ============ PegaRULES Bootstrap Configuration ============ Date: Tue Apr 28 18:19:04 AEST 2020 Java Version: Oracle Corporation 1.8.0_45 Configuration: file:/app/tomcat/tomcat4poc/webapps/prweb/WEB-INF/classes/prbootstrap.properties Pega-EngineCode: 08-04-02 Database: Oracle Oracle Database 12c Enterprise Edition Release Proprietary information hidden.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Advanced Analytics and Real Application Testing options Driver: Oracle JDBC driver Proprietary information hidden.0
(!) Unable to use given database configuration (!) Unable to locate classes on filesystem =============== End Bootstrap Configuration ===============
Unable to load phase 2 bootstrap class: com.pega.pegarules.internal.bootstrap.phase2.PRBootstrapImpl 28-Apr-2020 18:19:04.211 SEVERE [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrap. Error initializing PRAppLoader java.lang.ClassNotFoundException: com.pega.pegarules.internal.bootstrap.phase2.PRBootstrapImpl at com.pega.pegarules.internal.bootstrap.PRMiniLoader.loadClass(PRMiniLoader.java:378) at java.lang.ClassLoader.loadClass(Unknown Source) at com.pega.pegarules.internal.bootstrap.PRBootstrap.loadPhase2BootstrapClass(PRBootstrap.java:730) at com.pega.pegarules.internal.bootstrap.PRBootstrap.checkForStartup(PRBootstrap.java:673) at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:424) at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:225) at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:274) at com.pega.pegarules.internal.web.servlet.WebAppLifeCycleListenerBoot.contextInitialized(WebAppLifeCycleListenerBoot.java:92) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)
28-Apr-2020 18:19:04.211 SEVERE [localhost-startStop-1] com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega. Problem during method invocation (contextInitialized) java.lang.RuntimeException: PRAppLoader was not properly initialized, unable to load class com.pega.pegarules.web.servlet.WebAppLifeCycleListener at com.pega.pegarules.internal.bootstrap.PRBootstrap.getImplementationOf(PRBootstrap.java:222) at com.pega.pegarules.internal.bootstrap.PRBootstrap.getEntry(PRBootstrap.java:339) at com.pega.pegarules.internal.bootstrap.PRBootstrap.getMethod(PRBootstrap.java:319) at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:425) at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:225) at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:274) at com.pega.pegarules.internal.web.servlet.WebAppLifeCycleListenerBoot.contextInitialized(WebAppLifeCycleListenerBoot.java:92) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)
28-Apr-2020 18:19:04.212 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file 28-Apr-2020 18:19:04.212 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [/prweb] startup failed due to previous errors 28-Apr-2020 18:19:04.213 INFO [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrap. prbootstrap.properties merged with prbootstrap entries in Data-Admin-System-Settings 28-Apr-2020 18:19:04.213 INFO [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRMiniLoader. Will load phase 2 bootstrap from Pega-EngineCode:08-04-02 28-Apr-2020 18:19:04.222 WARNING [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRMiniLoader. com/pega/pegarules/internal/bootstrap/phase2/PRBootstrapImpl.class could not be loaded from PEGA_RULES_83.pr_engineclasses for '08-04-02' codeset. 28-Apr-2020 18:19:04.222 INFO [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRMiniLoader. Could not find file:/app/tomcat/tomcat4poc/webapps/prweb/WEB-INF/lib/prbootstrap-8.4.1-359.jar
===========================
Pegasystems Inc.
US
Hi Sreedhar,
You have taken care of several pointers and configuration looks fine to me.
The below message is a generic message whenever Pega fails to bootstrap from the provided DB details and won't give more info.
(!) Unable to use given database configuration (!) Unable to locate classes on filesystem
But I am interested in below error messages and can you verify
1. com/pega/pegarules/internal/bootstrap/phase2/PRBootstrapImpl.class could not be loaded from PEGA_RULES_83.pr_engineclasses for '08-04-02' codeset.
Can you confirm if the rules schema has the PRBootstrapImpl class definition with pega-enginecode codeset version as "08-04-02" ?
SELECT pzjar, pzpackage, pzclass, pzlastmodified, pzsha1, pzmoduleversion, pzcodeset, pzcodesetversion, pzpatchdate, pzstate, pzbinarycontent FROM 8.4_RULESCHEMA.pr_engineclasses where pzclass ='PRBootstrapImpl.class'
2. Could not find file:/app/tomcat/tomcat4poc/webapps/prweb/WEB-INF/lib/prbootstrap-8.4.1-359.jar
Can you also confirm if the prbootstrap-8.4.1-359.jar does exist in the above deployment directory under /prweb context? The exact jar files comes from 8.4.1 prweb.war archive file.
Let me know how it goes.
Transport for NSW
AU
Hi Harish,
Thanks for your reply.
1) I executed the query and it doesn't show any entry for 08-04-02.
Does it mean that upgrade didn't go fine? We didn't see any issues with upgrade as such in logs. First time we ran the upgrade script, it didn't respond for no reason after running for sometime without showing any errors. Then we re-initiated the upgrade (with resume.propoperties) and it went fine.
2) prbootstrap-8.4.1-359.jar is present in the specified location.
Pegasystems Inc.
US
Hi Sreedhar,
As a part of 8.4.1 update, the latest engine classes should be pushed into pr_engineclasses. There seems to be an issue as the query doesn't show-up 08-04-02 version of the PRBootstrapImpl class in pr_engineclasses.
Check if you have other class records in 08-04-02 codeset which are present in older codeset 08-03-xx. I strongly suspect the update was unsuccessful if you don't have 08-04-02 version of classes.
If Yes, you may need to Re-run the 8.4.1 upgrade and verify.
Transport for NSW
AU
We'll revert the schemas and re-run it. Thanks for your help.
Actually we've done one more upgrade Yesterday from 722 to 8.4 and it went fine. There I made a different observation in the post upgrade activities which we didn't see when we upgraded the same code base (722) to 8.3 earlier - Probably, I'll post a separate question for that.
Cognizant
GB
Hi Harish,
I have experienced the same issue, and the Pega is also on a similar configuration. I have performed both the checks you have specified and both (codeset and prbootstrap) were found as expected.
Is there anything else you can suggest to check?
Pratik Agarwal
Pegasystems Inc.
IN
Jdbc driver is fine. as per logs, get confirmation on below.
1. query the table "PEGA_RULES_83.pr_engineclasses" with includes codeset '08-04-02'& below class name in the where condition
"com/pega/pegarules/internal/bootstrap/phase2/PRBootstrapImpl.class could not be loaded from PEGA_RULES_83.pr_engineclasses for '08-04-02' codeset."
2. check the below path whether this jar is present "prbootstrap-8.4.1-359.jar"
"Could not find file:/app/tomcat/tomcat4poc/webapps/prweb/WEB-INF/lib/prbootstrap-8.4.1-359.jar"