Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 olivier K (olivierK7667)
Personal and Business Computers

Personal and Business Computers
FR
olivierK7667 Member since 2017 2 posts
Personal and Business Computers
Posted: Jan 5, 2018
Last activity: Feb 20, 2018
Posted: 5 Jan 2018 18:12 EST
Last activity: 20 Feb 2018 16:08 EST
Closed
Solved

Issue performance Deploying Pega Platform 7.3.1 on Microsoft Azure Cloud High availability

I built a high availability architecture on Microsoft Cloud Azure for using Pega platform 7.3.1 .

My architecture is as follows:

1 Azure Application Gateway => 2 Apache 2.4 Reverse Proxy Servers => 1 Azure Internal Loadbalancer => 2 Tomcat 7 Application Servers => 1 PostgreSql 9.6 Database Server ( I will use 2 servers for the futur : Master/Slave)


When the 2 tomcats servers are active, Pega team developers have some slowly or crashing while using the Pega prweb interface (tasks example : import project , Add a participants,quotation validation).


After several investigations, I noticed that by activating a single tomcat server, the navigation from the interface prweb pega is more fluid (more slowness, or chrash).


Note: I want to clarify that I do not have affinity session enabled at the Azure Internal Loadbalancer level because I only have 2 Affinity session options ( IP Client or Affinity session with Client Ip and Protocol )


However, in Pega's High Availability install documentation, it recommends configuring session affinity with cookie (which is not possible for Azure Internal Loadbalancer).

Show More

I built a high availability architecture on Microsoft Cloud Azure for using Pega platform 7.3.1 .

My architecture is as follows:

1 Azure Application Gateway => 2 Apache 2.4 Reverse Proxy Servers => 1 Azure Internal Loadbalancer => 2 Tomcat 7 Application Servers => 1 PostgreSql 9.6 Database Server ( I will use 2 servers for the futur : Master/Slave)


When the 2 tomcats servers are active, Pega team developers have some slowly or crashing while using the Pega prweb interface (tasks example : import project , Add a participants,quotation validation).


After several investigations, I noticed that by activating a single tomcat server, the navigation from the interface prweb pega is more fluid (more slowness, or chrash).


Note: I want to clarify that I do not have affinity session enabled at the Azure Internal Loadbalancer level because I only have 2 Affinity session options ( IP Client or Affinity session with Client Ip and Protocol )


However, in Pega's High Availability install documentation, it recommends configuring session affinity with cookie (which is not possible for Azure Internal Loadbalancer).

"Configuring affinity session
Affinity session is configured with the load balancer. It ensures that all requests from a user are handled
by the same Pega Platform server.
Production load balancers offer a range of options for configuring session affinity. Pega Platform supports
cookie-based affinity. Configure cookies for high availability session affinity using the following
variables.
Session / ha / quiesce / customSessionInvalidationMethod
Affinity session configuration is only applicable when using slow drain for quiesce.
l Value - configurable-cookie-invalidator or fully qualified class
SessionInvalidationMethod in prpublic
l Functionality - Class name of the invalidation method in use
Set to configurable-cookie-invalidator to perform cookie-based invalidation using the value
of session / ha / quiesce / cookieToInvalidate as the name of the cookie to invalidate
l Set up the interface of the interface
SessionInvalidationMethod in prpublic
Session / ha / quiesce / cookieToInvalidate
Affinity session configuration is only applicable when using slow drain for quiesce.
l Value - Name of the cookie to invalidate
Functionality - Name of the cookie to invalidate when using the configurable-cookie-invalidator
customSessionInvalidationMethod
Tier architecture application
Highly available tier architectures must have enough computing power to support fail-over,
more servers for support.
To deploy a highly available production system, two or more virtual machines are needed.
These allow for horizontal cluster
offer a solution for crash recovery. Each machine can deploy one or more Pega Platform servers.
Organizations can employ vertical cluster by adding multiple Pega Platform server instances to
production load support. Determine the number of the Pega Platform server instances based on capacity
planning and by consideration operational thresholds.
For information on support for server platform versions, see the Support Platform Guide on the
NDS. "

I have therefore made the following hypothesis concerning the origin of problem encounter :

- Session management concerns at the Azure Internal loadbalancer

- Concerns about high availability management (Azure loadbalancer / Pega application on Tomcat application server side)

- Or elsewhere in the connection chain (architecture)


Thank you in advance for your help, URGENT!


Show Less
To see attachments, please log in.
System Administration

  • Tanul Thanvi

Accepted Solution

Posted: 6 years ago
Posted: 7 Feb 2018 16:16 EST
olivier K (olivierK7667)
Personal and Business Computers

Personal and Business Computers
FR

The performances have now improved with the different tunning done:

- Tunning Apache Reverse Proxy (Timeout = 1200, ProxyTimeout = 1200, enable mod_deflate and setting on vhost file)

- Tunning Tomcat Server (Heapsize)
tomcat soft nofile 65535

tomcat hard nofile 65535

tomcat soft memlock unlimited

tomcat hard memlock unlimited

- Tunning PostGreSql Database server (work_mem, shared_buffer, etc.):

- Enable Session affinity on Azure gateway application (cookies) and Internal loadbalancer (IP Client)

- Set Parameters on prconfig.xml such as:
<env name = "alerts / browser / interactionTimeThreshold / warnMS" value = "1000" />
  <env name = "alerts / database / operationTimeThreshold" value = "250" />

Thank you for your help

To see attachments, please log in.
Posted: 6 years ago
Posted: 8 Jan 2018 4:21 EST
Naresh Reddy Tallapureddy (N@reshReddy) Senior Manager, Client Support Engineering - India
Pegasystems Inc.
IN

Hi,

I do understand that you have slowness even using single server, at this stage it may or may not be an issue with Session Affinity. For the slowness, you can contact Pega Support team with complete details and Pega-Alert log files to investigate and narrow down possible cause.

To comment on Session Affinity, an easy way to maintain affinity between a user and a server is to use user’s IP address when cookie based option is not available: this is called  IP affinity. This will “stick” a user to a server. It might have few issues and requires regression testing.

~Naresh

 

To see attachments, please log in.
Posted: 6 years ago
Posted: 8 Jan 2018 6:20 EST
olivier K (olivierK7667)
Personal and Business Computers

Personal and Business Computers
FR

Hi,

Thanks you for your response N@reshReddy ; I will check session affinity with Ip address .

I already checked this scenarios below :

a) run one server, but still under load balancer (via) => It’s whats I checked last friday,in fact with one tomcat server running ,belong to backend pool of  the Azure internal balancer : all things are good and don't have slowness

 

b) run a server (first one, then another) and connect directly, bypassing the LB =>  when I connect directly on tomcat URL  http://Tomcat_Server_Ip:8080 bypassing the LB :  all things are also good and don't have slowness

 

The next scenarios that I will test for the next day are as follows:

c ) Enable 1 or 2 tomcat servers by bypassing Azure Internal Lodbalancer:

We will have as architecture: 1 Azure Application Gateway => 2 Apache 2.4 Reverse Proxy Servers (with the module Loadbalancer of Apache active) => 2 Tomcat 7 Application Servers => 1 PostgreSql 9.6 Database Server (we will use 2 servers for the Database production environment: PostGreSql 9.6 Master / Slave

 

Show More

Hi,

Thanks you for your response N@reshReddy ; I will check session affinity with Ip address .

I already checked this scenarios below :

a) run one server, but still under load balancer (via) => It’s whats I checked last friday,in fact with one tomcat server running ,belong to backend pool of  the Azure internal balancer : all things are good and don't have slowness

 

b) run a server (first one, then another) and connect directly, bypassing the LB =>  when I connect directly on tomcat URL  http://Tomcat_Server_Ip:8080 bypassing the LB :  all things are also good and don't have slowness

 

The next scenarios that I will test for the next day are as follows:

c ) Enable 1 or 2 tomcat servers by bypassing Azure Internal Lodbalancer:

We will have as architecture: 1 Azure Application Gateway => 2 Apache 2.4 Reverse Proxy Servers (with the module Loadbalancer of Apache active) => 2 Tomcat 7 Application Servers => 1 PostgreSql 9.6 Database Server (we will use 2 servers for the Database production environment: PostGreSql 9.6 Master / Slave

 

d) Activate 1 or 2 tomcat servers by bypassing the Apache Reverse proxy servers and Azure Internal Loabalancer:

we will therefore have as architecture: 1 Azure Application Gateway => 2 Tomcat 7 Application Servers => 1 PostgreSql 9.6 Database Server (we will use 2 servers for the Database production environment: PostGreSql 9.6 Master / Slave)

 

Ps : If we need the current development environment to be stable and if possible improved performance or architecture or other advice are welcome (property depending on the Pega solution).

 

Thanks in advance for your help,


Show Less
To see attachments, please log in.
Posted: 6 years ago
Posted: 8 Jan 2018 8:58 EST
olivier K (olivierK7667)
Personal and Business Computers

Personal and Business Computers
FR

Correction :

1) single server = all is OK

2) multiple (i,e, 2) tomcat servers = issues:

-- when a developer changes BLA - I notice FOO ( for example import product,add a particpiant,

-- UI responding slow

-- In the  /usr/share/tomcat/work/Catalina/localhost/prweb

/PegaRULES-ALERT.log file , we have this error :

*Activity=Pega-UI-CommandPalette.pzGetPaletteOptions*Rule-Obj-Activity:pzGetPaletteOptions*PEGA-UI-COMMANDPALETTE PZGETPALETTEOPTIONS #20161123T194957.445 GMT Step: 2 Circum: 0*NA*****pxRDBIOElapsed=0.12;pxRDBIOCount=5;pxRunStreamCount=673;pxTotalReqCPU=1.97;pxRunModelCount=217;pxOutputBytes=478,017;pxRunWhenCount=1,565;pxDeclarativePageLoadElapsed=1.83;pxRulesExecuted=2,881;pxOtherCount=252;pxDBInputBytes=2,548,874;pxTotalReqTime=2.55;pxActivityCount=836;pxAlertCount=1;pxOtherFromCacheCount=73;pxRDBWithoutStreamCount=1;pxInteractions=1;pxLegacyRuleAPIUsedCount=1;pxRuleCount=220;pxInputBytes=101;pxRuleIOElapsed=0.00;pxRulesUsed=3,601;pxDeclarativePageLoadCount=6;pxRuleFromCacheCount=222;pxOtherIOElapsed=0.51;pxTrackedPropertyChangesCount=48;pxRDBRowWithoutStreamCount=1;pxOtherIOCount=196;*NA*NA*NA*NA*NA*pyActivity=Pega-UI-CommandPalette.pzGetPaletteOptions;*HTTP interaction has exceeded the elapsed time alert threshold of 1000 ms: 2546 ms.

Show More

Correction :

1) single server = all is OK

2) multiple (i,e, 2) tomcat servers = issues:

-- when a developer changes BLA - I notice FOO ( for example import product,add a particpiant,

-- UI responding slow

-- In the  /usr/share/tomcat/work/Catalina/localhost/prweb

/PegaRULES-ALERT.log file , we have this error :

*Activity=Pega-UI-CommandPalette.pzGetPaletteOptions*Rule-Obj-Activity:pzGetPaletteOptions*PEGA-UI-COMMANDPALETTE PZGETPALETTEOPTIONS #20161123T194957.445 GMT Step: 2 Circum: 0*NA*****pxRDBIOElapsed=0.12;pxRDBIOCount=5;pxRunStreamCount=673;pxTotalReqCPU=1.97;pxRunModelCount=217;pxOutputBytes=478,017;pxRunWhenCount=1,565;pxDeclarativePageLoadElapsed=1.83;pxRulesExecuted=2,881;pxOtherCount=252;pxDBInputBytes=2,548,874;pxTotalReqTime=2.55;pxActivityCount=836;pxAlertCount=1;pxOtherFromCacheCount=73;pxRDBWithoutStreamCount=1;pxInteractions=1;pxLegacyRuleAPIUsedCount=1;pxRuleCount=220;pxInputBytes=101;pxRuleIOElapsed=0.00;pxRulesUsed=3,601;pxDeclarativePageLoadCount=6;pxRuleFromCacheCount=222;pxOtherIOElapsed=0.51;pxTrackedPropertyChangesCount=48;pxRDBRowWithoutStreamCount=1;pxOtherIOCount=196;*NA*NA*NA*NA*NA*pyActivity=Pega-UI-CommandPalette.pzGetPaletteOptions;*HTTP interaction has exceeded the elapsed time alert threshold of 1000 ms: 2546 ms.

I found something on this link https://docs-previous.pega.com/pega0001-alert-http-interaction-time-exceeds-limit

To try to put this error as warning I put this line on my vim /var/lib/tomcat/webapps/prweb/WEB-INF/classes/prconfig.xml file :

 <env name="alerts/browser/interactionTimeThreshold/warnMS" value="1000" />

 

 


Show Less
To see attachments, please log in.
Posted: 6 years ago
Posted: 8 Jan 2018 22:27 EST
Naresh Reddy Tallapureddy (N@reshReddy) Senior Manager, Client Support Engineering - India
Pegasystems Inc.
IN

Its not suggested to increase the KPI values at this stage, as suggested earlier talk to Pega Support team/Azure Infra team on below.

1) Is there any delay reaching request LB? (Access.log or Http.log would help)

2) Is there any delay reaching request from LB to Server? (Can be checked with Pega HTTPAPI debug logs)

3) Is App Server sending delayed response to LB? Detailed investigation on Pega-Alert.log would help.

 

~Naresh

To see attachments, please log in.
Posted: 6 years ago
Posted: 9 Jan 2018 10:33 EST
olivier K (olivierK7667)
Personal and Business Computers

Personal and Business Computers
FR

Thank you for your help :

I enable  IP affinity on Azure Internal loadbalancer ( CLIENT IP ) : now  the connection is a little more fluid with the 2 tomcat servers actived :

Today , I noticed I Apache reverse proxy error file :

AH01102: error reading status line from remote server INTERNAL_IP:8080

Error reading from remote server returned by /prweb/PRPushServlet/!@514cca430f681a0b04a3fb34b03f6ca2!, referer: http://PRINCIPAL_URL/prweb/G3ADqXBQ1IuS_a_ZWCVlUw%5B%5B*/!STANDARD?pzPostData=1371192274
 

 

 

This is somes informations about servers , Application Gateway and LB parameters :

 

Show More

Thank you for your help :

I enable  IP affinity on Azure Internal loadbalancer ( CLIENT IP ) : now  the connection is a little more fluid with the 2 tomcat servers actived :

Today , I noticed I Apache reverse proxy error file :

AH01102: error reading status line from remote server INTERNAL_IP:8080

Error reading from remote server returned by /prweb/PRPushServlet/!@514cca430f681a0b04a3fb34b03f6ca2!, referer: http://PRINCIPAL_URL/prweb/G3ADqXBQ1IuS_a_ZWCVlUw%5B%5B*/!STANDARD?pzPostData=1371192274
 

 

 

This is somes informations about servers , Application Gateway and LB parameters :

 

------------------------------------------

Azure Application Gateway :

- For information about configuration :

Cookies based affinity : Disabled

Request timeout : 30

Protocol : HTTP

Port : 80


---------------------------------------------
Apache 2.4 Reverse Proxy :

- For information about configuration:

####### Apache.conf

Timeout 600

--------------------------------------------------------

Azure Internal Loadbalancer :

Port : 8080
Health probe : tcpProbe TCP 8080
Session persistence : Client IP
Idle timeout : 5 minutes

--------------------------------------------------------

Tomcat Server :

- For information about configuration : server.xml

    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" URIEncoding="UTF-8" />

-------------------------------------------------------------

PostGreSql Database servers :

max_connections = 100

shared_buffers = 128MB

work_mem = 5MB

autovacuum = on

 

 


Show Less
To see attachments, please log in.
Posted: 6 years ago
Posted: 23 Jan 2018 15:48 EST
olivier K (olivierK7667)
Personal and Business Computers

Personal and Business Computers
FR

Thank you for you precious help .


Indeed, since a few days, I did some tunning at the level of reverse proxy servers, tomcat servers and also the postgresql database servers : the application is much faster than before.

However, one of our developers has script batch that stop suddenly when they program at the Pega platform. I do not know what would cause this. 

Someone would have already met this problem?

Thank you in advance,

 

To see attachments, please log in.

Accepted Solution

Posted: 6 years ago
Posted: 7 Feb 2018 16:16 EST
olivier K (olivierK7667)
Personal and Business Computers

Personal and Business Computers
FR

The performances have now improved with the different tunning done:

- Tunning Apache Reverse Proxy (Timeout = 1200, ProxyTimeout = 1200, enable mod_deflate and setting on vhost file)

- Tunning Tomcat Server (Heapsize)
tomcat soft nofile 65535

tomcat hard nofile 65535

tomcat soft memlock unlimited

tomcat hard memlock unlimited

- Tunning PostGreSql Database server (work_mem, shared_buffer, etc.):

- Enable Session affinity on Azure gateway application (cookies) and Internal loadbalancer (IP Client)

- Set Parameters on prconfig.xml such as:
<env name = "alerts / browser / interactionTimeThreshold / warnMS" value = "1000" />
  <env name = "alerts / database / operationTimeThreshold" value = "250" />

Thank you for your help

To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice