too many open files
We've had two incidents in the past 2 weeks where we've run out of open files. We had increased the ulimit to 1,000,000 so it's leaking somewhere. Have other folks had this issue with Pega23?
***Edited by Moderator Marije. to add INC-B14495***
Accepted Solution
Updated: 17 May 2024 10:15 EDT
Pegasystems Inc.
GB
@DPCLARK12 the support ticket was closed with the following information:
Incident primary reason description:
PegaBroadcasterCache is getting overloaded because of so many verifydisconnectedRequestor push messages which will be send by server whenever client is idle for 40 secs. so, as a workaround we have asked client to lower the time the messages are stored in cache.
Solution description:
Below are the 2 DSS settings we can suggest client to lower their value
prconfig/server-push/message-cache/idle-time/default ( current value: 600, suggested value: 60)
prconfig/server-push/message-cache/prune-interval/default ( current value: 60, suggested value: 30)
Currently push notification messages are stored in cache for 600 secs( 10 min) which is long, so we will ask them to lower it to 1 min , so that cache won't be growing big in size.
Also, currently cache gets pruned for every 60 secs, so lets make it to 30 secs
These changes are tested in local and they are working as expected
Note: Client has to perform rolling restart after setting these DSS
Pegasystems Inc.
GB
@DPCLARK12 I see that no other forum users have experienced the symptoms you describe.
I did not find any other instances of the error in past support tickets.
Your support ticket is still open and relates to the pushservlet as your scenario occurs when files are loaded via FilesystemStorage. The org.atmosphere.util.SimpleBroadcaster is related and they are looking into helping you debug this issue further. It looks like the SimpleBroadcaster is pushing file to the server that are being read and causing OOM .
Feel free to continue your discussion with the GCS engineer who is liaising with SME's regarding that area of functionality.
Pegasystems Inc.
GB
@DPCLARK12 please can you contact our support team to confirm the results since their last call with you on April 30th:
suggested to update the below DSS as per below suggested values.
Please note that this change will require a rolling restart.
prconfig/server-push/message-cache/idle-time/default ( current value: 600, suggested value: 60)
prconfig/server-push/message-cache/prune-interval/default ( current value: 60, suggested value: 30)
Is it possible to do below changes and confirm ?
Cox Communications
US
@MarijeSchillern these are going in tomorrow morning
Accepted Solution
Updated: 17 May 2024 10:15 EDT
Pegasystems Inc.
GB
@DPCLARK12 the support ticket was closed with the following information:
Incident primary reason description:
PegaBroadcasterCache is getting overloaded because of so many verifydisconnectedRequestor push messages which will be send by server whenever client is idle for 40 secs. so, as a workaround we have asked client to lower the time the messages are stored in cache.
Solution description:
Below are the 2 DSS settings we can suggest client to lower their value
prconfig/server-push/message-cache/idle-time/default ( current value: 600, suggested value: 60)
prconfig/server-push/message-cache/prune-interval/default ( current value: 60, suggested value: 30)
Currently push notification messages are stored in cache for 600 secs( 10 min) which is long, so we will ask them to lower it to 1 min , so that cache won't be growing big in size.
Also, currently cache gets pruned for every 60 secs, so lets make it to 30 secs
These changes are tested in local and they are working as expected
Note: Client has to perform rolling restart after setting these DSS