We have a system where no one uses over the weekend. However our monitoring system detects the memory usage goes beyond the threshold a couple of times. I experienced some situation when Agents executes some operation at certain time like midnight in USA time, but this behavior does not seem to be coming from Agent execution (including custom / OOTB ) as it is some other random time. Does anyone have any idea that Pega is related to this or not. Or it is also helpful how to investigate what this memory increase is triggered out of.
Taking couple of manual Heap dumps during the time of High Memory usage together with the PegaRULES and PegaRULES-ALERT logs should help out pin-point the root cause / use case. Please ensure that if you are on Hotspot JRE you will need a manual heap dump generation utility (e.g. - JMAP) or if you are using IBM JRE then a IBM system dump. The following JVM arguments should help trigger a manual Heap dump.
you can produce a system dump instead on an user issued (kill -3 <pid> signal): You will need to add the following genericJvmArguments to your servers which are experiencing this issue.