System & Nodes going blank after Pods count increase
We are migrating our Pega Application(V 8.6.1) which is currently running as a Weblogic on Oracle DB setup to Openshift Container(on same Oracle DB). We successfully completed the migration. After migration, we want to increase the replicas of web pods. So we updated the pega yaml and redeployed the helm. Post deployment, System & nodes and Requestors landing pages are blank from Admin Studio. However could see the details of pods in pr_sys_statusnodes table showing recent pylastpulsedatetime. While tracing, found that step1(Map<String, String> clusterInfo = pega.getSystemOperationsProvider().getClusterManagementAPI().getClusterInformation(); ) of D_pzClusterMembersInformation is not populating pxNodeDetails. Please let me know of anything needs to be done to resolve this. Thanks!
***Edited by Moderator Marije to add Support Case Details INC-B12424 ***
Accepted Solution
Pegasystems Inc.
GB
@Jey2705 Ticket INC-B12424 has been resolved with the following final conclusion:
Explanation description:
The logs are indicating potential problems with insufficient resources causing instability. The below values are what is recommended for tuning the pods' resources effectively:
- CPU Req to 3 and Limit to 4, Memory Request and Limit both to 14GB, Min and Max Heap for Pega to 8GB
- Based on the YAML file that was provided, the current configuration will keep crashing nodes as heap is configured with same value as pod memory. Pod memory should be 30-40% more than Max heap.
- Also, any resource request is guranteed but limit is not...that is why memory request and memory limit need to remain at the same level.
cpuRequest: 3 memRequest: "14Gi" cpuLimit: 4 memLimit: "14Gi" initialHeap: "8000m" maxHeap: "8000m"
And regarding your second question:
@Jey2705 Ticket INC-B12424 has been resolved with the following final conclusion:
Explanation description:
The logs are indicating potential problems with insufficient resources causing instability. The below values are what is recommended for tuning the pods' resources effectively:
- CPU Req to 3 and Limit to 4, Memory Request and Limit both to 14GB, Min and Max Heap for Pega to 8GB
- Based on the YAML file that was provided, the current configuration will keep crashing nodes as heap is configured with same value as pod memory. Pod memory should be 30-40% more than Max heap.
- Also, any resource request is guranteed but limit is not...that is why memory request and memory limit need to remain at the same level.
cpuRequest: 3 memRequest: "14Gi" cpuLimit: 4 memLimit: "14Gi" initialHeap: "8000m" maxHeap: "8000m"
And regarding your second question:
-
Based on the documentation GCS found, it appears the JDK is included in the Pega provided docker image. The following article states, "Pega builds the latest Pega Platform Docker images, platform/pega and platform/installer, from a base image that includes Ubuntu 22, Java 11, and Tomcat 9." https://docs.pega.com/bundle/platform-88/page/platform/deployment/client-managed-cloud/pega-docker-images-manage.html#cmc-docker-images-overview
Pegasystems Inc.
GB
@Jey2705 I can see that you logged INC-B12424 (Post Upgrade ADMIN Studio Does not Show Nodes) and that our support team are looking into it.
Please post a reply once GCS resolves your issue, and provide the Accepted Solution here.
Pegasystems Inc.
GB
@Jey2705 INC-B12424 is still open and awaiting your response
Last update:
"As mentioned on the call, the logs are indicating potential problems with insufficient resources causing instability. The below values are what is recommended for tuning the pods' resources effectively:
- Set CPU Req to 3 and Limit to 4, Memory Request and Limit both to 14GB, Min and Max Heap for Pega to 8GB
- Based on the YAML file that was provided, the current configuration will keep crashing nodes as heap is configured with same value as pod memory. Pod memory should be 30-40% more than Max heap.
- Also, any resource request is guranteed but limit is not...that is why memory request and memory limit need to remain at the same level.
cpuRequest: 3 memRequest: "14Gi" cpuLimit: 4 memLimit: "14Gi" initialHeap: "8000m" maxHeap: "8000m"
Your team is planning to make these adjustments next week and test the startup and performance of the environment. Once this has been done, please update the GCS support team on the behavior you're observing post startup. ".
CSC
US
@MarijeSchillern Yes root cause has been identified as resource constraint with multiple namespace running in a single cluster. Our Admins are working to expand the infra before we deploy with the recommended resource setting.
Updated: 23 Apr 2024 9:33 EDT
CSC
US
@MarijeSchillern On a different note, another question came up during this process is the JDK version. is the JDK baked into Pega provided image already? Irrespective of the JAVA_HOME set during install or upgrade, will the application reference the JDK in the image(/opt/java/openjdk). Can you please confirm on that? Thanks!
Accepted Solution
Pegasystems Inc.
GB
@Jey2705 Ticket INC-B12424 has been resolved with the following final conclusion:
Explanation description:
The logs are indicating potential problems with insufficient resources causing instability. The below values are what is recommended for tuning the pods' resources effectively:
- CPU Req to 3 and Limit to 4, Memory Request and Limit both to 14GB, Min and Max Heap for Pega to 8GB
- Based on the YAML file that was provided, the current configuration will keep crashing nodes as heap is configured with same value as pod memory. Pod memory should be 30-40% more than Max heap.
- Also, any resource request is guranteed but limit is not...that is why memory request and memory limit need to remain at the same level.
cpuRequest: 3 memRequest: "14Gi" cpuLimit: 4 memLimit: "14Gi" initialHeap: "8000m" maxHeap: "8000m"
And regarding your second question:
@Jey2705 Ticket INC-B12424 has been resolved with the following final conclusion:
Explanation description:
The logs are indicating potential problems with insufficient resources causing instability. The below values are what is recommended for tuning the pods' resources effectively:
- CPU Req to 3 and Limit to 4, Memory Request and Limit both to 14GB, Min and Max Heap for Pega to 8GB
- Based on the YAML file that was provided, the current configuration will keep crashing nodes as heap is configured with same value as pod memory. Pod memory should be 30-40% more than Max heap.
- Also, any resource request is guranteed but limit is not...that is why memory request and memory limit need to remain at the same level.
cpuRequest: 3 memRequest: "14Gi" cpuLimit: 4 memLimit: "14Gi" initialHeap: "8000m" maxHeap: "8000m"
And regarding your second question:
-
Based on the documentation GCS found, it appears the JDK is included in the Pega provided docker image. The following article states, "Pega builds the latest Pega Platform Docker images, platform/pega and platform/installer, from a base image that includes Ubuntu 22, Java 11, and Tomcat 9." https://docs.pega.com/bundle/platform-88/page/platform/deployment/client-managed-cloud/pega-docker-images-manage.html#cmc-docker-images-overview