After importing log4J HFIX on 7.3,7.3.1 & 7.4 Pega Application went down
We have imported latest log4j hfix on 7.3 , 7.3.1 & 7.4, After restart pega engine is not coming up. Failed due to hazel cast errors.
Below are the Hfix installed in to the resp. pega system. { Our apps are still on 7.3 , 731 & 7.4}
|
7.3 |
|
DL-9012_HF-56098.zip |
|
7.3.1 |
|
DL-8670_HF-55820.zip |
|
7.4 |
|
DL-9013_HF-56099.zip |
We have imported latest log4j hfix on 7.3 , 7.3.1 & 7.4, After restart pega engine is not coming up. Failed due to hazel cast errors.
Below are the Hfix installed in to the resp. pega system. { Our apps are still on 7.3 , 731 & 7.4}
|
7.3 |
|
DL-9012_HF-56098.zip |
|
7.3.1 |
|
DL-8670_HF-55820.zip |
|
7.4 |
|
DL-9013_HF-56099.zip |
2022-01-05 01:54:59,476 [186238.devin3.ms.com] [ STANDARD] [ ] [ ] ( etier.impl.EngineStartup) ERROR - PegaRULES initialization failed. Server: ivapp1186238.devin3.ms.com com.pega.pegarules.pub.context.InitializationFailedError: PRNodeImpl init failed at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:404) ~[prprivate.jar:?] at com.pega.pegarules.session.internal.PRSessionProviderImpl.getThreadAndInitialize(PRSessionProviderImpl.java:2061) ~[prprivate.jar:?] at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineStartup.initEngine(EngineStartup.java:673) ~[prprivate.jar:?] at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._initEngine_privact(EngineImpl.java:172) ~[prprivate.jar:?] at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.doStartup(EngineImpl.java:145) ~[prprivate.jar:?] at com.pega.pegarules.web.servlet.WebAppLifeCycleListener._contextInitialized_privact(WebAppLifeCycleListener.java:326) ~[prwebj2ee.jar:?] at com.pega.pegarules.web.servlet.WebAppLifeCycleListener.contextInitialized(WebAppLifeCycleListener.java:221) ~[prwebj2ee.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_192] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_192] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_192] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_192] at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370) ~[prbootstrap-7.3.0-77.jar:7.3.0-77] at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411) ~[prbootstrap-7.3.0-77.jar:7.3.0-77] at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224) ~[prbootstrap-api-7.3.0-77.jar:7.3.0-77] at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:273) ~[prbootstrap-api-7.3.0-77.jar:7.3.0-77] at com.pega.pegarules.internal.web.servlet.WebAppLifeCycleListenerBoot.contextInitialized(WebAppLifeCycleListenerBoot.java:83) ~[prbootstrap-api-7.3.0-77.jar:7.3.0-77] at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4699) ~[catalina.jar:8.5.49] at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5165) ~[catalina.jar:8.5.49] at com.ms.webeng.tomcat.MorganStanleyContext.startInternal(MorganStanleyContext.java:204) ~[webinfra.tomcat-libs.jar:?] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[catalina.jar:8.5.49] at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743) ~[catalina.jar:8.5.49] at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719) ~[catalina.jar:8.5.49] at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705) ~[catalina.jar:8.5.49] at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:614) ~[catalina.jar:8.5.49] at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1823) ~[catalina.jar:8.5.49] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_192] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_192] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192] Caused by: java.lang.NoSuchFieldError: config at com.hazelcast.instance.EnterpriseNodeExtension.beforeStart(EnterpriseNodeExtension.java:150) ~[hazelcast-enterprise-3.10_1.jar:3.8] at com.hazelcast.instance.Node.<init>(Node.java:199) ~[hazelcast-3.8_1.jar:3.8] at com.hazelcast.instance.HazelcastInstanceImpl.createNode(HazelcastInstanceImpl.java:163) ~[hazelcast-3.8_1.jar:3.8] at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:130) ~[hazelcast-3.8_1.jar:3.8] at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:218) ~[hazelcast-3.8_1.jar:3.8] at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:176) ~[hazelcast-3.8_1.jar:3.8] at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:126) ~[hazelcast-3.8_1.jar:3.8] at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:58) ~[hazelcast-3.8_1.jar:3.8] at com.pega.platform.cluster.internal.HazelcastClusterProvider.initialize(HazelcastClusterProvider.java:109) ~[cluster-impl.jar:?] at com.pega.platform.cluster.internal.AbstractClusterProvider.start(AbstractClusterProvider.java:111) ~[cluster-impl.jar:?] at com.pega.platform.cluster.management.internal.DefaultClusterManager.start(DefaultClusterManager.java:114) ~[cluster-impl.jar:?] at com.pega.pegarules.cluster.internal.PRClusterHazelcastImpl.initialize(PRClusterHazelcastImpl.java:527) ~[prcluster.jar:?] at com.pega.pegarules.cluster.internal.PRClusterImpl.getInstance(PRClusterImpl.java:272) ~[prcluster.jar:?] at com.pega.pegarules.cluster.internal.PRClusterProviderImpl.getInstance(PRClusterProviderImpl.java:98) ~[prcluster.jar:?] at com.pega.pegarules.session.internal.mgmt.PRNodeImpl.joinCluster(PRNodeImpl.java:2715) ~[prprivate.jar:?] at com.pega.pegarules.session.internal.mgmt.PRNodeImpl.initializeSystem(PRNodeImpl.java:937) ~[prprivate.jar:?] at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:379) ~[prprivate.jar:?] ... 29 more 2022-01-05 01:54:59,485 [186238.devin3.ms.com] [ STANDARD] [ ] [ ] ( etier.impl.EngineStartup) INFO - PegaRULES initialization failed. Server: ivapp1186238.devin3.ms.com 2022-01-05 01:54:59,660 [186238.devin3.ms.com] [ STANDARD] [ ] [ ] (ervlet.WebAppLifeCycleListener) ERROR - Enterprise tier failed to initialize properly, PegaRULES not available 2022-01-05 01:54:59,673 [186238.devin3.ms.com] [ STANDARD] [ ] [ ] (ervlet.WebAppLifeCycleListener) INFO - Initializing push engine... 2022-01-05 01:55:03,604 [186238.devin3.ms.com] [ STANDARD] [ ] [ ] (tmosphere.AtmospherePushEngine) INFO - Async support implementation used by framework: org.atmosphere.container.JSR356AsyncSupport 2022-01-05 01:55:03,868 [186238.devin3.ms.com] [ STANDARD] [ ] [ ] (.PushSubscriptionCleanerDaemon) INFO - Push Subscription Cleaner Daemon initialized in web-tier. 2022-01-05 01:55:03,901 [186238.devin3.ms.com] [ STANDARD] [ ] [ ] (senceSessionStateTrackerDaemon) INFO - Presence Session State Tracker Daemon initialized in web-tier. 2022-01-05 01:55:03,901 [186238.devin3.ms.com] [ STANDARD] [ ] [ ] (ervlet.WebAppLifeCycleListener) INFO - Push engine initialization finished! 2022-01-05 01:55:03,901 [186238.devin3.ms.com] [ STANDARD] [ ] [ ] (ervlet.WebAppLifeCycleListener) INFO - Web Tier initialization is complete
***Edited by Moderator Marissa to add Support Case Details***
Pegasystems Inc.
NL
@HARIL043, you mean you've tried to install all 3 hotfixes on your Pega 7.4 environment?
If you're having critical production problems I always recommend reaching out to support immediately as well.
Updated: 6 Jan 2022 7:00 EST
MORGAN STANLEY
IN
@Eric Rietveld ..No .. in our org .. along with 8.x version, still few teams are using 7.3, 731 & 7.4 versions .. After importing Hfix in resp versions 7.3, 7.3.1 & 7.4. All these are failed with same hazelcast error.
yes already raised INC as well.
Updated: 7 Feb 2022 7:01 EST
W & W Informatik GmbH
DE
Did you resolve this enventually? May I ask how you did that?
We encountered a very similar issue today: After installing DL-9806 / HF-56881 (to address log4j vulnerability) on a 7.3.1 Pega system, all nodes failed to start.
Exception:
ERROR - PegaRULES initialization failed. Server: [servername] com.pega.pegarules.pub.context.InitializationFailedError: PRNodeImpl init failed
Caused by: java.lang.NoSuchFieldError: config
[...]
at com.hazelcast.instance.HazelcastInstanceImpl.createNode(HazelcastInstanceImpl.java:163) ~[hazelcast-3.8_1.jar:3.8]
Together with Pega support we figured out that apparently there were two parallel versions of Hazelcast (3.8 and 3.10) contained in database. After disabling the 3.8 classes, the system would start again.
However, these classes must have been in the DB for over a year and never caused a problem... Maybe the HFix somehow affected the class loading mechanism and thus caused Pega to load the older version vs. the newer? Pure speculation to this point.
Anyways, if any reader of this topic encounters the same issue, here is how we fixed it:
1. Check Hazelcast classes in Pega DB
Did you resolve this enventually? May I ask how you did that?
We encountered a very similar issue today: After installing DL-9806 / HF-56881 (to address log4j vulnerability) on a 7.3.1 Pega system, all nodes failed to start.
Exception:
ERROR - PegaRULES initialization failed. Server: [servername] com.pega.pegarules.pub.context.InitializationFailedError: PRNodeImpl init failed
Caused by: java.lang.NoSuchFieldError: config
[...]
at com.hazelcast.instance.HazelcastInstanceImpl.createNode(HazelcastInstanceImpl.java:163) ~[hazelcast-3.8_1.jar:3.8]
Together with Pega support we figured out that apparently there were two parallel versions of Hazelcast (3.8 and 3.10) contained in database. After disabling the 3.8 classes, the system would start again.
However, these classes must have been in the DB for over a year and never caused a problem... Maybe the HFix somehow affected the class loading mechanism and thus caused Pega to load the older version vs. the newer? Pure speculation to this point.
Anyways, if any reader of this topic encounters the same issue, here is how we fixed it:
1. Check Hazelcast classes in Pega DB
SELECT DISTINCT(pzclass), pzjar FROM RULES.PR_ENGINECLASSES WHERE (pzjar like '%hazelcast%' or pzclass like '%hazelcast%') and pzcodeset = 'pega-enginecode'
2. If there are multiple versions (see filename pzclass and / or package pzjar), update column enginecode of the unwanted version so Pega ignores these during class loading
# Example - disable 3.8:
UPDATE RULES.PR_ENGINECLASSES SET pzcodeset = 'pega.enginebackup' WHERE (pzjar like '%hazelcast%3.8%' or pzclass like '%hazelcast%3.8%')
3. Clear local temp directories on all Pega nodes
4. Restart Pega nodes
Disclaimer: Use this info at own risk; if production is affected, better raise a support request - that is also what we did!