Question
Last activity: 12 Jun 2017 12:33 EDT
Pega 7: how to configure hazelcast partion groups?
Hi,
I have a Pega 7 cluster which consists of 6 nodes. This cluster is used for a 24/7 application. The nodes are distributed across 2 datacenters. This provides a geographically redundant setup. Recently we had some issues with hazelcast during an upgrade. When investigating the issues, I found out that hazelcast has the option to work with partion groups. With the right definition of the hazelcast partition groups in place, hazelcast can continue to work, even when a datacenter goes down.
In order to achieve this, I need to tell hazelcast to put the nodes of 1 datacenter in a partion group and the nodes of the other datacenter in the another partion. For a standard java application this can be done by updating the configuration in the hazelcast jar file. Or by adding some java configuration code during initialization of the hazelcast API.
Pega does have several configuration opties for hazelcast, like port range to use, ip addresses to use and members of a hazelcast cluster. But I could not find a configuration option to define the hazelcast partion groups.
Does anyone have an idea how to configure hazelcast partition groups for the hazelcast that is used internally by Pega?
More info on hazelcast partition groups: http://docs.hazelcast.org/docs/3.2/manual/html-single/hazelcast-documentation.html#partition-group-configuration
Pegasystems Inc.
US
I see a prconfig setting in pega 7.2 - what is your pega version? I can check later if that setting is available there.
identification/cluster/configfile
where you can specify your custom hazelcast config file. I assume you know how to configure the partition group, similar to this link:
Rabobank
NL
When implemented Hazelcast configuration in Pega as mentioned in this thread, we run into a blocking issue.
Issue:
- When starting the PRPC after deploying the Hazelcast configuration as given below, PegaRULES initialization fails because of WrongTargetException from Hazelcast.
Additional info:
- Hazelcaset version 3.4.1
- Pega version 7.1.7 running on Linux+IBM WebShpere
- The PRPC can find the custom hazelcast config file via. prconfig.xml:
<env name="identification/cluster/configfile" value="<path to the file>/hazelcast-xxx.xml"/>
- The telnet for node1 on port 5701 is successful.
When implemented Hazelcast configuration in Pega as mentioned in this thread, we run into a blocking issue.
Issue:
- When starting the PRPC after deploying the Hazelcast configuration as given below, PegaRULES initialization fails because of WrongTargetException from Hazelcast.
Additional info:
- Hazelcaset version 3.4.1
- Pega version 7.1.7 running on Linux+IBM WebShpere
- The PRPC can find the custom hazelcast config file via. prconfig.xml:
<env name="identification/cluster/configfile" value="<path to the file>/hazelcast-xxx.xml"/>
- The telnet for node1 on port 5701 is successful.
- The curl request for the status of the hazelcast instance on the node shows that the config is picked up:
{
"request": {
"mbean": "com.hazelcast:instance=_hzInstance_1_dev,name=_hzInstance_1_dev,type=HazelcastInstance",
"type": "read"
},
"status": 200,
"timestamp": 1471858015,
"value": {
"Members": [
"/<ip of node 1>:5701"
],
"build": "20150213",
"clusterTime": 1471858015991,
"config": "Config{groupConfig=GroupConfig [name=dev, password=********], properties={}, networkConfig=NetworkConfig {publicAddress='null', port=5701, portCount=50, portAutoIncrement=true, join=JoinConfig{multicastConfig=MulticastConfig [enabled=false, multicastGroup= Proprietary information hidden, multicastPort=54327, multicastTimeToLive=32, multicastTimeoutSeconds=2, trustedInterfaces=[], loopbackModeEnabled=false], tcpIpConfig=TcpIpConfig [enabled=false, connectionTimeoutSeconds=5, members=[], requiredMember=null], awsConfig=AwsConfig{enabled=false, region='us-east-1', securityGroupName='null', tagKey='null', tagValue='null', hostHeader='ec2.amazonaws.com', connectionTimeoutSeconds=5}}, interfaces=InterfacesConfig{enabled=false, interfaces=[]}, sslConfig=null, socketInterceptorConfig=null, symmetricEncryptionConfig=null}, mapConfigs={default=MapConfig{name='default', inMemoryFormat=BINARY', backupCount=1, asyncBackupCount=0, timeToLiveSeconds=0, maxIdleSeconds=0, evictionPolicy='NONE', evictionPercentage=25, minEvictionCheckMillis=100, maxSizeConfig=MaxSizeConfig{maxSizePolicy='PER_NODE', size=2147483647}, readBackupData=false, nearCacheConfig=null, mapStoreConfig=null, mergePolicyConfig='com.hazelcast.map.merge.PutIfAbsentMapMergePolicy', wanReplicationRef=null, listenerConfigs=[], mapIndexConfigs=[]}}, topicConfigs={default=TopicConfig [name=default, globalOrderingEnabled=false]}, queueConfigs={}, multiMapConfigs={}, executorConfigs={default=ExecutorConfig{name='default', poolSize=16, queueCapacity=2147483647}}, semaphoreConfigs={}, wanReplicationConfigs={}, listenerConfigs=[], partitionGroupConfig=PartitionGroupConfig{enabled=true, groupType=CUSTOM, memberGroupConfigs=[MemberGroupConfig{interfaces=[node1]}, MemberGroupConfig{interfaces=[node2]}]}, managementCenterConfig=ManagementCenterConfig{enabled=false, url='null', updateInterval=3}, securityConfig=SecurityConfig{enabled=false, memberCredentialsConfig=CredentialsFactoryConfig{className='null', implementation=null, properties={}}, memberLoginModuleConfigs=[], clientLoginModuleConfigs=[], clientPolicyConfig=PermissionPolicyConfig{className='null', implementation=null, properties={}}, clientPermissionConfigs=[]}}",
"configSource": "<path to file>/hazelcast-xxx.xml",
"groupName": "dev",
"memberCount": 1,
"name": "_hzInstance_1_dev",
"port": 5701,
"running": true,
"version": "3.4.1"
}
}
- The current hazelcast config file looks like:
<?xml version="1.0" encoding="UTF-8"?>
<hazelcast>
<network>
<join>
<multicast enabled="false" />
</join>
<port port-count="50" auto-increment="true">5701</port>
</network>
<partition-group enabled="true" group-type="CUSTOM">
<member-group>
<interface>node1</interface>
</member-group>
<member-group>
<interface>node2</interface>
</member-group>
</partition-group>
</hazelcast>
- The error persist even when the hazelcast config is specified via. prconfig.xml:
<env name="cluster/hazelcast/multicast/enabled" value="false"/>
<env name="cluster/hazelcast/members" value="comma separated list of all nodes" />
<env name="cluster/hazelcast/ports" value="5701-5750" />
- Logs from Pega:
2016-08-22 07:19:26,244 GMT [zzzz] [ STANDARD] [ ] (nternal.PRClusterHazelcastImpl) INFO - Setting up Hazelcast with a custom cluster config file: <path to the file>/hazelcast-xxx.xml
2016-08-22 07:19:26,728 GMT [zzzz] [ STANDARD] [ ] (nternal.PRClusterHazelcastImpl) INFO - First node in this Cluster is starting.
2016-08-22 07:19:26,728 GMT [zzzz] [ STANDARD] [ ] ( internal.mgmt.PRNodeImpl) INFO - Clustering technology: hazelcast
2016-08-22 07:21:29,470 GMT [zzzz] [ STANDARD] [ ] ( etier.impl.EngineStartup) ERROR - PegaRULES initialization failed. Server: node1
com.pega.pegarules.pub.context.InitializationFailedError: PRNodeImpl init failed
at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:386)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.getThreadAndInitialize(PRSessionProviderImpl.java:1905)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineStartup.initEngine(EngineStartup.java:657)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._initEngine_privact(EngineImpl.java:165)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.doStartup(EngineImpl.java:138)
at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.doStartup(EngineBean.java:121)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
.
.
.
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984)
at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
Caused by:
com.hazelcast.spi.exception.WrongTargetException: WrongTarget! this:Address[<ip of node1>]:5701, target:null, partitionId: 8, replicaIndex: 0, operation: com.hazelcast.map.impl.operation.PutOperation, service: hz:impl:mapService
at com.hazelcast.spi.impl.BasicInvocation.initInvocationTarget(BasicInvocation.java:317)
at com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:253)
at com.hazelcast.spi.impl.BasicInvocation.run(BasicInvocation.java:614)
at com.hazelcast.spi.impl.BasicPartitionInvocation.run(BasicPartitionInvocation.java:28)
.
.
.
2016-08-22 07:21:29,471 GMT [zzzz] [ STANDARD] [ ] ( etier.impl.EngineStartup) INFO - PegaRULES initialization failed. Server: node1
2016-08-22 07:21:29,513 GMT [zzzz] [ STANDARD] [ ] (ervlet.WebAppLifeCycleListener) ERROR - Enterprise tier failed to initialize properly, PegaRULES not available
- Following discussion suggests that this issue is fixed long time ago but the error persists as shown above:
https://groups.google.com/forum/#!topic/hazelcast/fliSO75KaIo
I would appreciate any solutions, suggestions or ideas that may help in fixing this issue.
Thanks in advance.
Mandar K
PEG
IN
Could you please check the node entries in pr_sys_statusnodes table? Truncating the table and restarting the servers may help to resolve the issue.
Please refer https://community.pega.com/support/support-articles/many-references-hazelcast-errors-are-seen-logs
Pegasystems Inc.
US
Try removing the "<path to file>" text and probably the "/" as well in your configSource setting..
Remove "<ip of node 1>" from Member setting
Also - found these notes :
By setting "info" level logging for com.pega.pegarules.session.internal.mgmt.PRClusterImpl you can have PRPC report every state change to the logfile.
By setting prconfig setting "identification/cluster/logging" to "" you can have Hazelcast provide more verbose logging as well.