Question
Common Wealth Bank of Australia
AU
Last activity: 13 Sep 2018 17:18 EDT
Rule cache is not in sync in a multi node environment
We are using Pega 7.3.1 on tomcat with oracle DB. We have two nodes connected to the same DB. The rule updates done in node 1 is not reflected in node 2 & vice-versa
Verified that the Systempulse agent is up & running in both the nodes and there is no error in the logs. Also pr_sys_updatescache table in data schema is empty.
Any suggestion is highly appreciated.
Accepted Solution
Pegasystems Inc.
IN
Hazelcast cluster protocol is the default configuration used in Pega 7.x .
If you use IP address that would suffice the requirement for cluster just ports are need to open for the nodes communication with in the cluster .
You can try removing below configuration entries from the prconfig.xml and restart the nodes one by one .
<env name="cluster/hazelcast/multicast/enabled" value="false"/>
<env name="cluster/hazelcast/members" value="host name of node 1"/>
Thanks,
Arun
Pegasystems Inc.
IN
Are you sure pysystemname is same for two nodes in pr_sys_statusnodes table(in data schema)?
The systemnames of the two nodes must be same in order to be in the same cluster and to maintain rule sync.
Sytem pulse will no longer be used to maintain rule sync among nodes when using hazelcast/ignite clustering technology but is used to indicate that a node is up whcih intern is used by hazelcast/ignite to maintain rule sync among nodes.
pr_sys_statusupdates table will be empty when using clustering protocols/technologies to maintain rule sync.
Please make sure the sytemnames of two nodes are same.
Common Wealth Bank of Australia
AU
Hi Adithya,
The pysystemname value is same for both nodes (with the value pega) in the table pr_sys_statusnodes. Also pyclusteraddress has value nodeIP:5701 & pyrunstate is running for both the nodes in the table pr_sys_statusnodes.
I could not see any table with the name pr_sys_statusupdates. Do you mean pr_sys_statusdetails ? If yes, then the table pr_sys_statusdetails is not empty and contains more than 5000 records.
Also is there any specific configuration for hazelcast that i need verify eg: prconfig.xml. In my environment I have the below settings in prconfig.xml with regards to hazelcast
NODE 1:
<env name="cluster/hazelcast/interface" value="IP address of node 1"/>
<env name="cluster/hazelcast/multicast/enabled" value="false"/>
<env name="cluster/hazelcast/members" value="host name of node 1"/>
NODE 2:
<env name="cluster/hazelcast/interface" value="IP address of node 2"/>
<env name="cluster/hazelcast/multicast/enabled" value="false"/>
<env name="cluster/hazelcast/members" value="host name of node 2"/>
Pegasystems Inc.
IN
Sorry for the typo in my previous comment. I was referring to ps_sys_updatescache.
It seems ports and members for hazelcast cluster are all set in nodes.
Make sure you don't have this in any of the node
Sorry for the typo in my previous comment. I was referring to ps_sys_updatescache.
It seems ports and members for hazelcast cluster are all set in nodes.
Make sure you don't have this in any of the node
<env name=”identification/cluster/protocol” value=”none”/>
Virtusa IT Consulting
AE
I believe you are using incorrect values for the two prconfig settings you shared,
<env name=”identification/cluster/protocol” value=”none”/> - I am not sure who recommended this to you, usually you should not touch this and you have made it to none, so I am afraid how hazelcast will be able play its role here.
<env name="cluster/hazelcast/members" value="host name of node 1"/> - you have mentioned this setting separately for the two nodes, so I don't think both the members will be a part of one cluster, however are you able to see the two members getting added in the PegaRULES log?
For example:
member[2]{
member_1:5701 --> This
member_2:5702
}
Also, Have you tried using the setting as <env name="cluster/hazelcast/members" value="host name of node 1,host name of node 2"/> ?
Accepted Solution
Pegasystems Inc.
IN
Hazelcast cluster protocol is the default configuration used in Pega 7.x .
If you use IP address that would suffice the requirement for cluster just ports are need to open for the nodes communication with in the cluster .
You can try removing below configuration entries from the prconfig.xml and restart the nodes one by one .
<env name="cluster/hazelcast/multicast/enabled" value="false"/>
<env name="cluster/hazelcast/members" value="host name of node 1"/>
Thanks,
Arun
Common Wealth Bank of Australia
AU
Hi Arun, Thanks for the response. After removing those settings from the prconfig.xml , the rule sync up issue is resolved.
In addition to the above change in the prconfig.xml , i restarted the nodes one by one and not simultaneously (which is different from our environment deployment pattern).
Our environment setup runs pega inside the docker container.So our deployment pattern will deploy the pega docker containers simultaneously on both the nodes. Does this pattern of parallel deployment of docker containers onto multiple VMs will cause any issue in Hazelcast configuration /Elastic search ?
Centene
US
Karthigeyan
When the docker starts the container image, its may assign a different port then how does the other pega cluster understand that the hazelcast is actually running on a different port?
In my environment, we have 2 dockers server running 3 containers each. So the synchronizing is not happening because when the server start docker is assigning a different port for each container. So what configurations can be done to fix the issue. It would be great if you can share the sample prconfig.xml file from any one node.
Right now search is not working for us in any of the nodes because of hazelcast(I believe)
Your response is greatly appreciated.
Pegasystems Inc.
IN
Hello Karthi,
In your case nodes were not communicating with each other which caused the rule sync issue .
So in order to avoid these scenarios you can restart the nodes one by one so that it will try to join the cluster and communicate with each other .
Also part of docker container deployment do you clear Pega Temp folder ?
Hello Karthi,
In your case nodes were not communicating with each other which caused the rule sync issue .
So in order to avoid these scenarios you can restart the nodes one by one so that it will try to join the cluster and communicate with each other .
Also part of docker container deployment do you clear Pega Temp folder ?
Thanks,
Arun
Common Wealth Bank of Australia
AU
Hi Arun,
Apologies for the very late response. We normally do not delete the Pega temp folder as part of the docker deployment.