Applies to Pega Platform versions 7.2.1 through 8.6

This document is one in a series that includes the following companion documents:

Managing clusters with Hazelcast

Split-Brain Syndrome and cluster fracturing FAQs

Troubleshooting Hazelcast cluster management 

Important: See the latest Pega Documentation cited in the Related Content section of this document.

This document identifies improvements provided in the Pega Platform™ for Hazelcast editions.

Beginning with Pega Platform 8, support for Hazelcast Community Edition is replaced by support for Hazelcast Enterprise Edition.

Keeping current with Pega and Hazelcast

Hazelcast 3.10 EE with Pega 8.1 and later releases

Hazelcast 3.11.5 with Pega 8.1 and later releases

Hazelcast 4.x with Pega 8.6 and later releases

Pega 8.6 Alerts: PEGA0125, PEGA0126, PEGA0127

Keeping current with Pega and Hazelcast

Best Practice: Keep current with Pega

To obtain a hotfix, follow these steps:

1. Go to My Support Portal.
2. Click Create a ticket.
3. In Create a new support ticket, click Request > Existing hotfix.
4. Complete all fields of the form and Submit it.

Pega Platform supports the Hazelcast Editions shown in the table below.

Hazelcast 3.10 EE with Pega 8.1 and later releases

Enterprises using Pega 7.1.x need to upgrade to Pega 8.1 and later releases to benefit from the Hazelcast 3.10 Enterprise Edition (EE) perpetual license and stability improvements in the Pega Platform. Here are the details about the critical improvements that are provided:

Critical importance for all Pega 7 users

• Environment updates to Hazelcast 3.10 Enterprise Edition (EE) because Hazelcast 3.8 Community Edition (CE) is no longer supported
• A perpetual license for Hazelcast Enterprise Edition (until 2099)
• Several Pega Cluster changes centered around stability and resiliency
• Logging, alert, and event improvements for Clustering and Hazelcast
• A variety of API changes

Hazelcast 3.11.5 with Pega 8.1 and later releases

Enterprises using Pega 8.1 and later releases need to take advantage of Hazelcast 3.11.5 and the improvements described in this section:

• Updates customers to use Hazelcast 3.11.5 Enterprise Edition

High importance for all Pega 8 users

Provided in Pega 8.4 and later releases, Pega 8.3.2, Pega 8.2.6, Pega 8.2.5, Pega 8.1.8

• Fixes an issue in Hazelcast where nodes, starting at the same time as the Master Node is stopped, fail to start

High importance for all Pega 8 users

Provided in Pega 8.4 and later releases, Pega 8.3.2, Pega 8.2.6, Pega 8.2.5, Pega 8.1.8

• Handles changing Hazelcast instance UUIDs
  • Upon split-brain recovery, Hazelcast UUID changes are now accounted for in the Cluster code.
  • Without the Pega Platform Patch Releases or hotfixes, nodes that recover from split-brain might not be able to communicate effectively. This leads to a propagation of failures across the cluster.

Critical importance for all Pega users

Provided in Pega 8.4 and later releases, Pega 8.3.1, Pega 8.2.5, Pega 8.2.3 HFix-57475, Pega 8.1.7, Pega 8.1.6 HFix-57381, Pega 8.1.5 HFix-58684, Pega 7.4 HFix-60101, Pega 7.3.1 HFix-57586

• Admin Studio improvements (remote execution usage)

The Admin Studio landing pages are sub-optimally used for the cluster to fetch data. For large clusters, it is possible for the page to freeze indefinitely if the cluster is under duress. This change ensures that the Pega engine and Admin Studio are responsive in the wake of issues with the cluster.

Very high importance for all Pega 8 users

Provided by Pega 8.4 and later releases, Pega 8.3.2, Pega 8.3.1 HFix-59970, Pega 8.2.6, Pega 8.2.5 HFix-58486, Pega 8.1.8

• Decision Strategy Manager (DSM) discovery of node topology leads to unresponsiveness.

Updated code fixes an issue that causes a node to request information from all nodes, causing nodes to be stuck in topology discovery forever.

In smaller clusters (less than 15 nodes), slowness is observed, but the cluster remains stable.

In larger clusters (15 nodes or more), a race condition leads to a distributed deadlock.

Critical importance for all Pega 8.1 users

Provided by Pega 8.2 and later releases, Pega 8.1.4, Pega 8.1.3 HFix-51612, Pega 8.1.2, Pega 8.1.1, and Pega 8.1.0

• Building Cache Manager multiple times and HazelcastCacheBuilder race condition
  • Cache Manager is now a singleton object to prevent it from being created twice.
  • A race condition in Hazelcast code where multiple threads creating the same cache leads to an error is now fixed.

High importance for Pega 7 users

Provided by Pega 8.1 and later releases, Pega 7.4.17, Pega 7.4.16 HFix-60090, Pega 7.4, Pega 7.3.1, Pega 7.3

• Decision Data Store (DDS) Service checks for candidate nodes.

Hazelcast is involved even when the DDS service is configured against an external Cassandra cluster. Generic DSM service code checks for candidate nodes, that is, nodes possibly available for DSM services. This behavior has been in effect since Pega 7.3. Updated code ensures that DDS service codes do not check for candidate nodes when an external cluster is configured.

High importance for all Pega 8 users

Provided by Pega 8.4, Pega 8.3.2, Pega 8.2.6, Pega 8.2.5 HFix-59894, Pega 8.1.8

• Lookups for application requestors are made to Operator Presence.

Operator Presence does not store data on application requestors. Updated code removed the call made to Operator Presence to look up application requestors.

Important for Pega 7 and Pega 8 users

Provided by Pega 8.1 and later releases, Pega 7.4 HFix-57282, Pega 7.3.1 and earlier releases

• Hazelcast 3.11.5  startup fails.
  • Starting nodes do not find the master node because the master node shuts down within the default time out. Updated code increases the default timeout to 10 minutes.
  • Starting nodes fail to start if listener registration fails or times out. Updated code gracefully handles exceptions during listener registration.

High importance for all Pega 8 users

Provided by Pega 8.6, Pega 8.5.1, Pega 8.4.3, Pega 8.3.4

• PegaInstaller is unable to join the cluster.
      Installation succeeds, but Import fails because nodes are unable to join the cluster.
      Pega 8.1.8 Cloud 2.15.5   

• Frequent nodes are terminated because of Hazelcast errors.
  OperationTimeoutException or QueryPartitionOperationQueryPartitionOperation for presence-related operation appear in thread dumps. The following message is also displayed:
  "The message exceeds recommended size of 8000"

Important for all Pega users

See Pega 7.3.1 SA-68067 Frequent nodes getting terminated due to Hazelcast error.

Alternative Solution: Add the common operator that has many sessions associated with it to the deny list.

• Admin Studio gets blocked when multiple nodes are restarted simultaneously.

Important for users of Pega 8.2.6 and later releases

Provided by Pega 8.2.6 HFix-68735 and HFIX-70408, Pega 8.4.5, Pega 8.5.4, and Pega 8.6.1

• Out of Memory (OOM) errors occur on a regular basis.
  Updated code fixes memory leaks in the remote execution framework.

Fix is planned for Pega 8.6.1, Pega 8.5.3, and Pega 8.4.5.

Perform the following local change:
Disable pyPersistNodeState and pyPersistClusterState jobs
 

Hazelcast 4.x with Pega 8.6 and later releases

This is not a default version of Hazelcast for Pega Platform on-premises deployments. During installation and upgrade, Hazelcast 4.x EE  requires an additional prconfig setting:

Prconfig value = “cluster/hazelcast/v4/enabled” = "true”

Upgrading to Hazelcast 4.x requires downtime. Therefore, consider which option to use:

• Keep the current Hazelcast version enabled during your installation of or upgrade to Pega 8.6.
• Install or upgrade to Pega 8.6 first and then take downtime to upgrade to Hazelcast 4.x.

Pega 8.6 Alerts: PEGA0125, PEGA0126, PEGA0127

Nodes do not show in Admin Studio even though they are up and running and can be seen in the table pr_sys_statusnodes. Because of a temporary database error, after the Service registry sets the status as safe mode, the cluster never recovers unless the node is restarted.

Updated code fixes this problem and documents new alerts:

HEARTBEAT_FAILED("PEGA0125", "Service registry heartbeat has failed"),

ENTERING_SAFE_MODE("PEGA0126", "Service registry is entering safe mode"),

RECOVERED_SAFE_MODE("PEGA0127", "Service registry recovered from safe mode");

Important for users of Pega 8.4 and later releases           

Provided by Fixed in 8.4.3, 8.5.1, 8.6

See the following articles:

PEGA0125 alert: Service registry heartbeat failed

PEGA0126 alert: Service registry is entering safe mode

PEGA0127 alert: Service registry recovered from safe mode

Related Content

PEGA0090 alert: Hazelcast partition was lost

External Hazelcast in your deployment

Connecting to your managed Hazelcast service

Configuring communication between the Pega cluster and the Hazelcast cluster

Setting the outbound port for cluster communication