Admin studio doesn't show up any nodes information.
As I said in title, we can't see any nodes and overview in admin studio often. our on-premise cluster.
I understand this is fixed as long as restarting pega tomcat.
But I want to know the cause not to occur like this.
I noticed in log below, does anyone how about this ?
rationsDistributedExecutorImpl) ERROR - Caught exception while getting result from future object (perations.ClusterManagementAPI) ERROR - exception while fetching high level node info for the node c579ea6
Pegasystems Inc.
GB
@MasashiY error "exception while fetching high level node info for the node" is a sign that there are network issues between the nodes.
You may be seeing HZ split brain and database disconnections pointing to network connection issues. Other symptoms could be HTTP Pool threads stuck doing hostname lookup
1. Check OS Ulimits and make them persistent following IBM Support article: https://www.ibm.com/support/pages/node/151329
2. Shutdown cluster cleaning all pega temp folders
3. Update JVM arguments to disable IPv6 and hostname lookup following IBM Support article: https://www.ibm.com/support/pages/slow-performance-or-hang-hostname-lookup Relevant arguments reported below -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true
4. Restart all the Servers, ensuring no port binding issues occurs and double check that there no incidents during startup. In case of any issues, address them and restart the full cluster again.
Users in the past were able to resolve it by modifying the etc/hosts file at linux level.
Please see the troubleshooting steps outlined in this post "Server is up but Node is not in running state in Admin Studio."
@MasashiY error "exception while fetching high level node info for the node" is a sign that there are network issues between the nodes.
You may be seeing HZ split brain and database disconnections pointing to network connection issues. Other symptoms could be HTTP Pool threads stuck doing hostname lookup
1. Check OS Ulimits and make them persistent following IBM Support article: https://www.ibm.com/support/pages/node/151329
2. Shutdown cluster cleaning all pega temp folders
3. Update JVM arguments to disable IPv6 and hostname lookup following IBM Support article: https://www.ibm.com/support/pages/slow-performance-or-hang-hostname-lookup Relevant arguments reported below -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true
4. Restart all the Servers, ensuring no port binding issues occurs and double check that there no incidents during startup. In case of any issues, address them and restart the full cluster again.
Users in the past were able to resolve it by modifying the etc/hosts file at linux level.
Please see the troubleshooting steps outlined in this post "Server is up but Node is not in running state in Admin Studio."
Other possible causes: pega nodes are dropping off from the cluster after 30 odd minutes
---> Could you sum up the connection pool for each tomcat node against the db connection pool size.
Ex: If the DB connection pool size set to 500 and there are 6 tomcat nodes and each node has 100 connection pool size then it may lead to the instability in the connection pool and nodes will be dettached from the cluster with out any reason.
Other cause suggested: Caused when different nodes being in different timezones w.r.t DB timezone. Once, all the timezones were brought in sync, the issue could be resolved.
Application server config settings: see this post.
If you require this to be investigated in more details then we would suggest that you log a support incident for this.
In the ticket you will be asked to provide:
1. PegaRULES, PegaALERT, PegaCLUSTER, GC logs and the prconfig from all the nodes in the cluster. 2. Kafka server logs from all stream nodes. 3. Current list of installed hot fixes. 4. List of JVM parameters configured for the nodes in the cluster. 5. Data source configuration for the nodes in the cluster. 6. Is Pega PDC configured in the environment having the issue? If yes then please share the instance name that we can review. 7. Do you see any heap dump getting generated in the nodes having the issue? 8. If you have any server level monitoring, is it possible to share any CPU/Memory utilization graphs over last 2-3 days of period from the environment covering the time of last restart. 9. Total Memory and CPU allocated for each node.
If you decide to log a support ticket, please can you provide the INC number here so that we can help track it?