Applies to Pega Platform™ versions 8.5.3 through 8.8.
Symptoms
Production nodes are intermittently going down due to high CPU and heap memory utilization. Production nodes are down for an unacceptable time. From the Heap dump, you see the PassivationDaemon Thread occupying 6.6 GB of memory. You observe the regular exceptions and later see that thread dumps created later throw the OOM exception with JVM down.
Errors
Out-of-Memory exception
Explanation
The DirectStreamEncoder default setting holding memory is causing heap dump utilization resulting in nodes going down. Heap dump analysis shows the DirectStreamEncoder objects are retaining 7G of heap causing resource contention.
The default behavior is to return the encoder instance to the connection pool as-is. Here, the size of the pool can grow based on the page that you encode, even though the number of encoders remain the same. The resulting high page size causes the node to go down.
Details
The DirectStreamEncoder default value takes advantage of the following connection pooling benefits:
- It boosts the performance of the application significantly.
- It is most effective in a situation where the rate of initializing a class instance is high.
- It manages the connections and provides a way to reuse and share them.
- It can also provide the limit for the maximum number of objects that can be created.
Cases reported for the DirectStreamEncoder pool indicate that DirectStreamEncoder objects in the pool can increase in size and cause memory issues when trying to encode Clipboard pages having large amounts of data.
This raises the question:
What value should be used for prconfig in the file prconfig.xml or in the DSS clipboard/encoder/poolConfig ?
Solutions
To prevent this issue from occurring, open the file prconfig.xml in edit mode and add the following parameter:
<env name="clipboard/encoder/poolConfig" value="reset-contents" />
Restart all nodes in the cluster.
The reset-contents parameter resets the contents of the DirectStreamEncoder object before returning the instance to the pool. This means that all ArrayList and byte[] references are reset to their default size like 8 for ArrayList and 1024 for byte[].
If specifying the reset-contents parameter in the file prconfig.xml does not resolve the issue, then set the value of the following DSS:
Owning ruleset: Pega-Engine
Purpose: prconfig/clipboard/encoder/poolConfig/default
Value: no-pool
Restart all nodes in the cluster.
This DSS might cause some side effects like losing the advantages of the connection pooling mechanism.
This DSS clears contents of the Encoder that is returned to the pool. Therefore, the Encoder that goes to the pool is leaner. The Encoder object retains the structure of the Page encoded last time.
Related content
Product Documentation
Creating a dynamic system setting
Configuring the Stream service
