Question
Royal Bank of Canada
CA
Last activity: 22 Sep 2020 6:01 EDT
Potential Load Balancing Issue with Incoming Requests
Hi,
We have been facing a sudden Requester Spike issue in Pega application where one JVM will be affected with huge number of requesters in a short time period and CPU spike high in production.
We are thinking it is issue with load balancer distribution across all JVM's as we have many pega applications running across all 4 nodes with load balancer.
1.Is there any Configuration/setting that can be checked in Load Balancer side or Web Server side for more information?
2.Is there anything that we configure in Pega for requestors passing through load balancer and to Webserver JVMS?
3.Security Integration in the web container is a default setting as it pretty much guarantees that a user HTTP Session is mapped to the correct user when accessing secured pages to protect the session from unauthorized access.
What is relation between the “Requestor’s count and a user HTTP session because we are using that as the meter to compare the load distribution.
Please help us on solving this issue.Thank you.
Regards,
Geetha
***Edited by Moderator Marissa to update General to Product***
Pegasystems Inc.
US
Can you share the details of the surging requestors? I would go to Admin Studio first. Are these browser requestors or something else? Also if you can share the topology of your env, that would help as well. I assume your production env has been running for a while. If that is the case, are there any recent changes in the environment?
Updated: 16 Jun 2020 0:00 EDT
Royal Bank of Canada
CA
Its Browser requestors and you see below from log message requestor count increasing from 132 to 526 and to 1042 within 10minutes and this requestor count is spiking suddenly on 1 JVM and CPU spikes high due to this requestor count.
On any regular day, the requestors count is 100-150 requestors.
Server details: We have 4 web servers and 4 application servers in cluster environment.
Each 4 app servers(node 1,2,3,4) has 3 JVMs each(node 1-1,node 1-2,node 1-3...) in cluster env.
Its Browser requestors and you see below from log message requestor count increasing from 132 to 526 and to 1042 within 10minutes and this requestor count is spiking suddenly on 1 JVM and CPU spikes high due to this requestor count.
On any regular day, the requestors count is 100-150 requestors.
Server details: We have 4 web servers and 4 application servers in cluster environment.
Each 4 app servers(node 1,2,3,4) has 3 JVMs each(node 1-1,node 1-2,node 1-3...) in cluster env.
2020-06-02 09:14:14,780 [.PRPCWorkManager : 3] [ STANDARD] [ ] [ PegaAPPli:02.01.02] ( internal.async.Agent) INFO - System date: Tue Jun 02 09:14:14 EDT 2020 Total memory: 8,544,059,392 Free memory: 4,011,364,232 Requestor Count: 132 Shared Pages memory usage: 0% Current number of threads: 8 2020-06-02 09:24:14,783 [.PRPCWorkManager : 0] [ STANDARD] [ ] [ PegaAPPli:01.01.01] ( internal.async.Agent) INFO - System date: Tue Jun 02 09:24:14 EDT 2020 Total memory: 8,544,059,392 Free memory: 2,912,995,160 Requestor Count: 526 Shared Pages memory usage: 0% Current number of threads: 13
Application is in production env from 4 years and Pega version is PRPC 7.4V
Proximus
BE
Hello,
for me above issue is not linked to load balancing.
It can either be linked to a lock on pr_operator and a lock a DB level.
Why so many requestors? I think it's because incoming requests are still received but then cannot perform/commit their action.
So the requestor is not released and as such PEGA opens a new requestor for the new incoming request.
You can quickly see a Requestor Count pretty high if your processing is not releasing previous requestors.
Since it's going on one JVM only above, I would suspect a issue in this node only.
for more investigations, we would need more info on what's doing this node.
Are you facing the same issue?
Regards
Anthony