Question
Macquarie Bank
AU
Last activity: 2 Jun 2023 7:42 EDT
Service REST Performance Issue
Hi All,
We have created a Service REST in Pega 8.5.2. We ran some performance testing before we move the code to PROD. during our testing we have identified some performance issues, as we can see some spikes in response time during the runs. Usually the expected response time is between 150-300 milliseconds.
But for few requests it is taking 1500-3500 milliseconds.
We are currently using LDAP authentication in the service package. When I tried to debug this, I have noticed that below steps took more time.
- com.pegarules.generated.activity.ra_action_authenticationldapverifycredentials_acbd1556cf6cf403197e19e3463b35ea.step2_circum0
- java.util.concurrent.locks.LockSupport.park
- com.pega.platform.executionengine.vtable.ruleres.VirtualTableCandidate.getExecutableContainerFor
- com.pega.pegarules.exec.internal.util.classloader.PRClassLoaderDB$ClassDefinition.getElement
- om.pega.pegarules.exec.internal.util.classloader.PRClassLoaderDB$ClassDefinition:getElement Blocked on com.pega.pegarules.exec.internal.util.classloader.PRClassLoaderDB$ClassDefinition@3e7dd63d
I have changed it to Basic Authentication as well to see if any improvement in performance. But I am seeing more time is taking at this step now.
- com.pega.platform.securitycore.encryption.internal.EncryptionImpl.samePasswordAndUpgradeIfRequired
Can you please help me in understanding this and how I can improve my performance of the API.
***Edited by Moderator Marije to add Support Case Details; ***
Accepted Solution
Pegasystems Inc.
GB
I can see that the support ticket was resolved with the following solution:
Root cause description:
Query on How to achieve performance bench mark for the business use case.
Solution type: Explanation
Understanding behavior of Java threads and related performance KPI:
- The KPI observed for any transaction would vary based on multiple underlying factor such as OS state, the number of active thread on JVM and many more.
- Since we are considering very fine margin (fraction of seconds) a little change on OS and JVM variable can cause API thread to execute with variable KPI. The same thread that takes 300 ms can take 600 ms
- It would not be entirely possible to get consistent response time to a precision of few 100 ms for 100% of the transactions. There would be outliers.
- In the worst case scenario, with 15 API all running with slowest KPI of 1.2 Seconds would mean around 17 seconds for a user. However Probability of this happening would be very very low (close to zero percent, 1/15*15)
Understanding why is the external LDAP has average execution 170 Ms less than the Pega authentication:
I can see that the support ticket was resolved with the following solution:
Root cause description:
Query on How to achieve performance bench mark for the business use case.
Solution type: Explanation
Understanding behavior of Java threads and related performance KPI:
- The KPI observed for any transaction would vary based on multiple underlying factor such as OS state, the number of active thread on JVM and many more.
- Since we are considering very fine margin (fraction of seconds) a little change on OS and JVM variable can cause API thread to execute with variable KPI. The same thread that takes 300 ms can take 600 ms
- It would not be entirely possible to get consistent response time to a precision of few 100 ms for 100% of the transactions. There would be outliers.
- In the worst case scenario, with 15 API all running with slowest KPI of 1.2 Seconds would mean around 17 seconds for a user. However Probability of this happening would be very very low (close to zero percent, 1/15*15)
Understanding why is the external LDAP has average execution 170 Ms less than the Pega authentication:
- Pega engine and LDAP are two different entity and has different logic purpose to serve.
- The LDAP server primary purpose is to validate credential and it is slightly efficient than Basic authentication that Pega provided.
JWT authentication was found to be more efficient for the business use case.
Client decided to use JWT Authentication as an alternative to Basic/LDAP Auth for the API's. There was significant improvement in the Avg response time (response time is 50-60% less than Basic/LDAP Auth) for each API when using JWT as Auth Mechanism.
JWT authentication will be implemented without sizing the existing infrastructure.
Pegasystems Inc.
GB
Can you please search for the below access group : Access Group : PRPC:Unauthenticated Access Role : PegaRULES:Guest
Please check the execute activities level set to 5 or not for the Unauthenticated access group.
In 8.x releases we added some enhancements in fetching access token through rest endpoints flow. We are expecting Operator page for getting some attributes in PegaROPCGrantHandler class but that page is being removed after executing the activity pyCustomIdentityMappingAct as part of one cleanup step in AuthenticationLDAPVerifyCredentials which is being internally called. There is no issue with web login , so the solution is users need to add Page-Copy step in pyCustomIdentityMappingAct activity if they want to invoke rest endpoint for token.
The documentation for Service REST can be found here.
Also please take a look at the Resolved Issues in 8.5.3.
Also check the Support Document Pega Platform 8.6.6 Patch Resolved Issues
There have been many fixes relating to OAuth and authentication issues.
Can you please search for the below access group : Access Group : PRPC:Unauthenticated Access Role : PegaRULES:Guest
Please check the execute activities level set to 5 or not for the Unauthenticated access group.
In 8.x releases we added some enhancements in fetching access token through rest endpoints flow. We are expecting Operator page for getting some attributes in PegaROPCGrantHandler class but that page is being removed after executing the activity pyCustomIdentityMappingAct as part of one cleanup step in AuthenticationLDAPVerifyCredentials which is being internally called. There is no issue with web login , so the solution is users need to add Page-Copy step in pyCustomIdentityMappingAct activity if they want to invoke rest endpoint for token.
The documentation for Service REST can be found here.
Also please take a look at the Resolved Issues in 8.5.3.
Also check the Support Document Pega Platform 8.6.6 Patch Resolved Issues
There have been many fixes relating to OAuth and authentication issues.
It would be prudent to test your scenario again on 8.5.3. If you still need further help I suggest that you log a support incident for this. Please can you let us know the INC reference so that we can help track it?
Macquarie Bank
AU
@MarijeSchillern Thanks for your response Marije. I checked the Access Role PegaRULES:Guest has execute activities level as 5 . Please refer to attachment.
Even when I changed the service package authentication to Basic, we are still seeing spikes in the response time. can you please share any link to understand the sequential steps of basic authenticate execution when a service call is invoked. I tried to trace from the requestor, but didn't find any rules executing this below step.
- com.pega.platform.securitycore.encryption.internal.EncryptionImpl.samePasswordAndUpgradeIfRequired
Updated: 11 Jan 2023 5:10 EST
Pegasystems Inc.
GB
@NavakanthMannem our documentation can be found here
Configuring login using a custom authentication service
Defining processing and authentication for service packages
On Pega Academy:
If you still need further help I suggest that you log a support incident for this. You can do so on the MSP.
Please can you let us know the INC reference so that we can help track it?
Macquarie Bank
AU
@MarijeSchillern can you please elaborate more on adding Page-Copy step in pyCustomIdentityMappingAct activity. I could find this activity is not being invoked from AuthenticationLDAPVertifyCredentials. And also we are not fetching any access token from rest endpoint.
Thanks, Navakanth M.
Pegasystems Inc.
GB
@NavakanthM16711766
If you still need further help I suggest that you log a support incident for this. You can do so on the MSP.
Please can you let us know the INC reference so that we can help track it?
Macquarie Bank
AU
@MarijeSchillern Thanks for your response.
I have created a support incident (INC-255783) for the same. Please let me know if this helps.
Accepted Solution
Pegasystems Inc.
GB
I can see that the support ticket was resolved with the following solution:
Root cause description:
Query on How to achieve performance bench mark for the business use case.
Solution type: Explanation
Understanding behavior of Java threads and related performance KPI:
- The KPI observed for any transaction would vary based on multiple underlying factor such as OS state, the number of active thread on JVM and many more.
- Since we are considering very fine margin (fraction of seconds) a little change on OS and JVM variable can cause API thread to execute with variable KPI. The same thread that takes 300 ms can take 600 ms
- It would not be entirely possible to get consistent response time to a precision of few 100 ms for 100% of the transactions. There would be outliers.
- In the worst case scenario, with 15 API all running with slowest KPI of 1.2 Seconds would mean around 17 seconds for a user. However Probability of this happening would be very very low (close to zero percent, 1/15*15)
Understanding why is the external LDAP has average execution 170 Ms less than the Pega authentication:
I can see that the support ticket was resolved with the following solution:
Root cause description:
Query on How to achieve performance bench mark for the business use case.
Solution type: Explanation
Understanding behavior of Java threads and related performance KPI:
- The KPI observed for any transaction would vary based on multiple underlying factor such as OS state, the number of active thread on JVM and many more.
- Since we are considering very fine margin (fraction of seconds) a little change on OS and JVM variable can cause API thread to execute with variable KPI. The same thread that takes 300 ms can take 600 ms
- It would not be entirely possible to get consistent response time to a precision of few 100 ms for 100% of the transactions. There would be outliers.
- In the worst case scenario, with 15 API all running with slowest KPI of 1.2 Seconds would mean around 17 seconds for a user. However Probability of this happening would be very very low (close to zero percent, 1/15*15)
Understanding why is the external LDAP has average execution 170 Ms less than the Pega authentication:
- Pega engine and LDAP are two different entity and has different logic purpose to serve.
- The LDAP server primary purpose is to validate credential and it is slightly efficient than Basic authentication that Pega provided.
JWT authentication was found to be more efficient for the business use case.
Client decided to use JWT Authentication as an alternative to Basic/LDAP Auth for the API's. There was significant improvement in the Avg response time (response time is 50-60% less than Basic/LDAP Auth) for each API when using JWT as Auth Mechanism.
JWT authentication will be implemented without sizing the existing infrastructure.