IAC Memory Leak
Hi,
We are facing memory leaks in the IAC server that is running on Tomcat.
I've attached a file with the current JVM arguments.
Attached few more images to give a picture of what's happening.
We would like to know how to tune the JVM arguments to perform optimally. We think there might be a problem with the garbage collector.
Updated: 23 Mar 2020 12:50 EDT
Pegasystems Inc.
US
Hi Sujith,
Do you have any application other than IAC(prgateway) deployed on this server? If Yes, you should look at their usage too.
Capturing a heap dump will tell you where the maximum amount of heap memory is accumulated and if there is a leak you can identify by looking at the number of objects for the same type and whether they are supposed to be cleaned up or not etc .. And writing GC logs will tell you how GC is functioning.
Based on your application usage & issue bottleneck, you can play around with adding other JVM arguments to fine-tune the performance.
But first and foremost, I would like to suggest the below settings to generate heap dump on OOM error and write GC logs which helps in RCA.
-XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path to dump>
Let me know if that clarifies.
IQZ Systems LLC
US
Hi Harish,
The heap dump is more than 6GB and most of the content are just gibberish, it's not easy to understand. I can share the dump file to you if you have an FTP.
I have attached the gc log here.
Have you checked the JVM arguments I have attached, is there anything that seems off or wrong that we can remove or add something that you think might improve?
Regards,
Sujithram
Pegasystems Inc.
US
Analyzing heap dumps takes a significant amount of time. I would recommend creating an SR.
GC logs showed no signs of memory leak and you may use https://gceasy.io/ and upload GC log to understand how many minor & major GC's happened.
In general, you may include the below JVM args as well.
-XX:+UseStringDeduplication
IQZ Systems LLC
US
Hi Harish,
I tried -XX:+UseStringDeduplication, but that didn't make it better. All the nodes are working fine except for one. We have the same arguments in all the nodes, but just one particular node uses a lot of memory. There was no thread dump in the path specified, however the memory is being utilized at 97%.
This environment is inactive, there is no network traffic whatsoever, but still there is a high memory usage.
I assumed this could be an environmental issue. Do you think this assumption is right?
From your experience, do you know anything that could be causing this just on one node? Any help would be much appreciated.
We can raise an SR when we can confirm this is actually an IAC issue, because if we mention that it is only on one node, they might come back and ask us to check the environment.
If you can think of anything that could cause this anomaly, please do let us know.
Pegasystems Inc.
US
Do you have any application other than IAC(prgateway) deployed on this server? If Yes, you should look at their usage too.
The other nodes also have the same no of applications deployed?
When you say"This environment is inactive, there is no network traffic whatsoever, but still there is high memory usage." I don't think there will be any Pega platform issue, but looking at heap dump will confirm which objects are eating up the memory.
I can look into the HeapDump if I find some time next week, but I can help to isolate things and won't be able to recommend anything if the problem is outside scope of Pega.
https://sftpcam.pega.com/public/folder/__L9jBZQ502uSwdex6J8Ug/HeapDump
pw: pega123