Discussion
Pegasystems Inc.
IT
Last activity: 2 Jul 2020 2:49 EDT
Data store and stream nodes not joining on Oracle WebLogic 12.2.1.3 after Patch Set Update
With recent Oracle JDK versions (JDK 7 Update 131 or later/JDK 8 Update 121 or later) and WebLogic Server October 2017 PSU or later, WebLogic implements JEP 290 support which increases security on RMI (see Oracle Fusion Middleware - Configuring a Custom JEP 290 Deserialization Filter).
After starting the Cassandra JVM, the Pega Platform tries to connect on local Cassandra JMX port (default 7199) to control the instance.
With WebLogic default security configuration, despite Cassandra JVM being correctly started and the JMX port being open and reachable (note it is local and bound to localhost), the Platform will indefinitely wait for connection to the JMX port to successfully complete.
In the WebLogic managed server log, the following message will show every second:
####<Mar 14, 2019 5:00:08,827 PM CET> <Info> <java.io.serialization> <***> <***> <DSM init service: DDS.Default> <<anonymous>> <> <9115d352-aa60-436c-aedf-66c65cc4ea3f-0000000d> <1552575608827> <[severity-value: 64] [rid: 0:3:1:1:8] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <ObjectInputFilter REJECTED: class java.rmi.server.RemoteObject, array length: -1, nRefs: 6, depth: 3, bytes: 169, ex: n/a>
To allow the Platform to successfully control the Cassandra instance on its local JMX port, add the following to the Pega Platform JVM parameters:
-Dweblogic.oif.serialFilterMode=combine -Dweblogic.oif.serialFilter=java.rmi.server.RemoteObject
Note that the class java.rmi.server.RemoteObject is the one shown in the log message.
Similarly, after starting the Kafka JVM, the Pega Platform tries to connect on local Kafka JMX port (default 9999) to control the instance. In this case, add the following to the Pega Platform JVM parameters:
-Dweblogic.oif.serialFilterMode=combine -Dweblogic.oif.serialFilter=java.rmi.server.RemoteObjectInvocationHandler;java.rmi.server.RemoteObject
-
Your friendly neighborhood Technical Architect
Updated: 27 Mar 2020 22:58 EDT
Liberty Global PLC
GB
Thank you for this ..This solved the problem and saved my day !
After applying WLS patch : Patch 30675853: WLS PATCH SET UPDATE Proprietary information hidden.191217(ID:191217.1425) | Release Date : 14-Jan-2020
Cassandra and Kafka process took ages to startup, the node kept crashing on start up was showing up with warning in the Weblogic console with stuck threads
Admin studio was not showing the nodes that were assinged DDS, ADM, Stream, Search in Running state
Dev studio it was either showing as Joining / Joining Failed for DDS , ADM, Stream .
WebUser nodes were failing with the below error as well
'com.pega.dsm.dnode.impl.cassandra.DdsLoggingHelper$UpdateQueryLoggersJob' was unable to be completed. (com.hazelcast.spi.exception.RetryableHazelcastException) HazelcastInstance[[IP]:PORT] is not active!
(.PresenceLifeCycleEventManager) ERROR <hostname>|<ip> operatorid - Failed to update session state for requestor with ID: <Requestor ID>
com.hazelcast.core.HazelcastInstanceNotActiveException: State: SHUT_DOWN Operation: class com.hazelcast.map.impl.query.QueryPartitionOperation
After adding the 2 JVM arguments mentioned above to all nodes it worked like a charm. Thanks again.
Might help if the SA-88049 could be amended to reflect the same , missing out on java.rmi.server.RemoteObjectInvocationHandler
Thank you for this ..This solved the problem and saved my day !
After applying WLS patch : Patch 30675853: WLS PATCH SET UPDATE Proprietary information hidden.191217(ID:191217.1425) | Release Date : 14-Jan-2020
Cassandra and Kafka process took ages to startup, the node kept crashing on start up was showing up with warning in the Weblogic console with stuck threads
Admin studio was not showing the nodes that were assinged DDS, ADM, Stream, Search in Running state
Dev studio it was either showing as Joining / Joining Failed for DDS , ADM, Stream .
WebUser nodes were failing with the below error as well
'com.pega.dsm.dnode.impl.cassandra.DdsLoggingHelper$UpdateQueryLoggersJob' was unable to be completed. (com.hazelcast.spi.exception.RetryableHazelcastException) HazelcastInstance[[IP]:PORT] is not active!
(.PresenceLifeCycleEventManager) ERROR <hostname>|<ip> operatorid - Failed to update session state for requestor with ID: <Requestor ID>
com.hazelcast.core.HazelcastInstanceNotActiveException: State: SHUT_DOWN Operation: class com.hazelcast.map.impl.query.QueryPartitionOperation
After adding the 2 JVM arguments mentioned above to all nodes it worked like a charm. Thanks again.
Might help if the SA-88049 could be amended to reflect the same , missing out on java.rmi.server.RemoteObjectInvocationHandler
Updated: 26 May 2020 10:17 EDT
Corporation Service Company
US
Observed similar issue with different versions:
Pega Version: 8.4
Java: jdk1.8.0_251
Weblogic: Proprietary information hidden
Error during start up:
<Info> <java.io.serialization> <system> <pegaserver_1> <DSM init service: StreamServer.Default> <<anonymous>> <> <23935918-fe9f-4477-b9fb-ec6680eaaf53-00000009> <1590501180622> <[severity-value: 64] [rid: 0:17] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <ObjectInputFilter REJECTED: class java.rmi.server.RemoteObjectInvocationHandler, array length: -1, nRefs: 7, depth: 2, bytes: 213, ex: n/a>
This is not resolved after setting the JVM arguments mentioned in that article. Please suggest if we need to perform any other changes