List of Nodes running changes to only 1 after a time
Hi All,
We are running Pega 8.6.2 and in our SIT environment we have a 2 node cluster. During startup one node with show both nodes running in Admin Studio. In Decisioning > Infrastructure > Services > Stream, both nodes are always shown with status NORMAL clicking either node will show the relevant details.
But after a short while the remote node will disappear from the node list with nothing showing in the logs either PegaRULES or PegaCLUSTER to indicate why. However stream will still show both nodes as NORMAL but clicking on the remote node shows nothing and produces an error in the PegaRULES log. The Search landing page also shows both nodes as ONLINE.
I've tried all sorts of prconfig settings to get around this problem to no avail. Below are the settings I've tried.
<env name="identification/cluster/public/address" value="xxx.xxx.xxx.xxx"/>
<env name="cluster/hazelcast/interface" value="xxx.xxx.xxx.xxx"/>
<env name="cluster/hazelcast/members" value="xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx"/>
Table pr_sys_statusnodes always shows both nodes correctly.
PegaCLUSTER log always shows the 2 nodes becoming members of the cluster and shows the relevant node being removed when that node is shutdown.
Ports are open between both nodes and even when the remote host disappears from the node list telnet between hosts on those ports still functions.
Anyone got any idea as to what might be causing this?
Hi All,
We are running Pega 8.6.2 and in our SIT environment we have a 2 node cluster. During startup one node with show both nodes running in Admin Studio. In Decisioning > Infrastructure > Services > Stream, both nodes are always shown with status NORMAL clicking either node will show the relevant details.
But after a short while the remote node will disappear from the node list with nothing showing in the logs either PegaRULES or PegaCLUSTER to indicate why. However stream will still show both nodes as NORMAL but clicking on the remote node shows nothing and produces an error in the PegaRULES log. The Search landing page also shows both nodes as ONLINE.
I've tried all sorts of prconfig settings to get around this problem to no avail. Below are the settings I've tried.
<env name="identification/cluster/public/address" value="xxx.xxx.xxx.xxx"/>
<env name="cluster/hazelcast/interface" value="xxx.xxx.xxx.xxx"/>
<env name="cluster/hazelcast/members" value="xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx"/>
Table pr_sys_statusnodes always shows both nodes correctly.
PegaCLUSTER log always shows the 2 nodes becoming members of the cluster and shows the relevant node being removed when that node is shutdown.
Ports are open between both nodes and even when the remote host disappears from the node list telnet between hosts on those ports still functions.
Anyone got any idea as to what might be causing this?
We are trying to work with Quiesce for HA but this problem is preventing us from testing.
Thanks Craig
***Edited by Moderator Marije to add Support Case Details; update capability tags***
Virgin Media
GB
Hi thanks for the response, no I'm not seeing any exceptions in PDC for this. Which is why it's extremely annoying.
I managed to get 1 node to remain showing both nodes by removing Stream from it's node types, but the other simply only shows its own node still.
Thanks
Pegasystems Inc.
GB
Please see the troubleshooting steps outlined in this post "Server is up but Node is not in running state in Admin Studio."
Also see Admin studio doesn't show up any nodes information
Other possible causes: pega nodes are dropping off from the cluster after 30 odd minutes
---> Could you sum up the connection pool for each node against the db connection pool size.
Ex: If the DB connection pool size set to 500 and there are 6 nodes and each node has 100 connection pool size then it may lead to the instability in the connection pool and nodes will be detached from the cluster with out any reason.
Other cause suggested: Caused when different nodes being in different timezones w.r.t DB timezone. Once, all the timezones are brought in sync, the issue may be resolved.
Other Application server config settings: https://collaborate.pega.com/question/node-not-showing-admin-studio
If you require this to be investigated in more details I would suggest that you log a support incident for this.
In the ticket you will be asked to provide:
Please see the troubleshooting steps outlined in this post "Server is up but Node is not in running state in Admin Studio."
Also see Admin studio doesn't show up any nodes information
Other possible causes: pega nodes are dropping off from the cluster after 30 odd minutes
---> Could you sum up the connection pool for each node against the db connection pool size.
Ex: If the DB connection pool size set to 500 and there are 6 nodes and each node has 100 connection pool size then it may lead to the instability in the connection pool and nodes will be detached from the cluster with out any reason.
Other cause suggested: Caused when different nodes being in different timezones w.r.t DB timezone. Once, all the timezones are brought in sync, the issue may be resolved.
Other Application server config settings: https://collaborate.pega.com/question/node-not-showing-admin-studio
If you require this to be investigated in more details I would suggest that you log a support incident for this.
In the ticket you will be asked to provide:
1. PegaRULES, PegaALERT, PegaCLUSTER, GC logs and the prconfig from all the nodes in the cluster.
2. Kafka server logs from all stream nodes.
3. Current list of installed hot fixes.
4. List of JVM parameters configured for the nodes in the cluster.
5. Data source configuration for the nodes in the cluster.
6. Is Pega PDC configured in the environment having the issue? If yes then please share the instance name with our support team.
7. Do you see any heap dump getting generated in the nodes having the issue?
8. If you have any server level monitoring, is it possible to share any CPU/Memory utilization graphs over last 2-3 days of period from the environment covering the time of last restart.
9. Total Memory and CPU allocated for each node.
If you decide to log a support ticket, please can you provide the INC number here so that we can help track it?
Virgin Media
GB
I've already been through those questions and none have helped. Database Connection pool size is 1000 and each of the 2 nodes is configured with 100 max connections so that isn't the issue.
For reference we are running Oracle and Tomcat on Linux. It's the only environment we seem to be having this issue and I can't explain why. As mentioned there is nothing in the PegaCLUSTER.log file when the node disappears to indicate any kind of problem. Last entry for the affected node shows only this
2022-05-01 13:49:29,303 [c-operation.thread-0] [ ] [ ] (nternal.cluster.ClusterService) INFO - [xxx.xxx.xxx.10]:5701 [2edcbc2999c5a8d5573885a736ceacd4] [4.1.1]
Members {size:2, ver:18} [
Member [xxx.xxx.xxx.11]:5701 - fa413d55-3e37-4aa0-9145-1fb6cb406c4c
Member [xxx.xxx.xxx.10]:5701 - 654ba658-13fa-4e3b-956f-937b7d221aa3 this
]
2022-05-01 13:49:30,262 [ dcmpega-kn-t11] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxx.xxx.xxx.10]:5701 [2edcbc2999c5a8d5573885a736ceacd4] [4.1.1] [xxx.xxx.xxx.10]:5701 is STARTED
And at the same time this is the log entry for the node now not affected after removing Stream node type.
I've already been through those questions and none have helped. Database Connection pool size is 1000 and each of the 2 nodes is configured with 100 max connections so that isn't the issue.
For reference we are running Oracle and Tomcat on Linux. It's the only environment we seem to be having this issue and I can't explain why. As mentioned there is nothing in the PegaCLUSTER.log file when the node disappears to indicate any kind of problem. Last entry for the affected node shows only this
2022-05-01 13:49:29,303 [c-operation.thread-0] [ ] [ ] (nternal.cluster.ClusterService) INFO - [xxx.xxx.xxx.10]:5701 [2edcbc2999c5a8d5573885a736ceacd4] [4.1.1]
Members {size:2, ver:18} [
Member [xxx.xxx.xxx.11]:5701 - fa413d55-3e37-4aa0-9145-1fb6cb406c4c
Member [xxx.xxx.xxx.10]:5701 - 654ba658-13fa-4e3b-956f-937b7d221aa3 this
]
2022-05-01 13:49:30,262 [ dcmpega-kn-t11] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxx.xxx.xxx.10]:5701 [2edcbc2999c5a8d5573885a736ceacd4] [4.1.1] [xxx.xxx.xxx.10]:5701 is STARTED
And at the same time this is the log entry for the node now not affected after removing Stream node type.
2022-05-01 13:49:29,242 [c-operation.thread-0] [ ] [ ] (nternal.cluster.ClusterService) INFO - [ Proprietary information hidden]:5701 [2edcbc2999c5a8d5573885a736ceacd4] [4.1.1]
Members {size:2, ver:18} [
Member [xxx.xxx.xxx.11]:5701 - fa413d55-3e37-4aa0-9145-1fb6cb406c4c this
Member [xxx.xxx.xxx.10]:5701 - 654ba658-13fa-4e3b-956f-937b7d221aa3
]
2022-05-01 13:49:29,576 [goldwasser.migration] [ ] [ ] (artition.impl.MigrationManager) INFO - [xxx.xxx.xxx.11]:5701 [2edcbc2999c5a8d5573885a736ceacd4] [4.1.1] Repartitioning cluster data. Migration tasks count: 271
2022-05-01 13:49:31,194 [goldwasser.migration] [ ] [ ] (artition.impl.MigrationManager) INFO - [xxx.xxx.xxx.11]:5701 [2edcbc2999c5a8d5573885a736ceacd4] [4.1.1] All migration tasks have been completed. (repartitionTime=Sun May 01 13:49:29 BST 2022, plannedMigrations=271, completedMigrations=271, remainingMigrations=0, totalCompletedMigrations=678)
So clearly they know that each of them are there.
Pegasystems Inc.
GB
@CraigA52 did you check the article I referenced where for another user the solution was related to a setting on the etc/hosts file at linux level?
Please log a support incident and let us know the INC here so that we can help track this.
Virgin Media
GB
@MarijeSchillernINC-223436 raised as nothing I've attempted has made any difference.