Applies to all major, minor, and patch releases of Pega Platform™ versions 8.2 through 8.7

Symptoms
Errors
Explanations
Environments
Solutions
Related content

Symptoms

A variety of scenarios reported very slow performance with JVM startup, application server startup, application startup, including exception errors for Pega bootstrap, FUA, and SQL timeouts.

Scenario 1 Pega Platform version 8.6.5 JVM startup takes 20 minutes after update from Pega Platform version 7.2.2
Scenario 2 Application server takes almost two hours to start after update to Pega Platform version 8.7.4
Scenario 3 Application takes over an hour to start after update to a newer Pega Platform version 8 release
Scenario 4 Server startup takes 75 minutes on Pega Platform version 8.4.6
Scenario 5 Issue connecting Oracle 19C database from Pega Platform version 8.2
Scenario 6 Slow restart of services after installing the hotfix to address the Log4j vulnerability
Scenario 7 Application server restart of Postgres RDS takes longer than expected
Scenario 8 Pega Platform version 8.5.3 takes a long time to start up

Scenario 1 Pega Platform version 8.6.5 JVM startup takes 20 minutes after update from Pega Platform version 7.2.2

After successfully updating your Pega Platform™ version 7.2.2 to Pega Platform version 8.6.5, you notice that the Pega system starts but two JVMs take 20 minutes to restart.

Scenario 2 Application server takes almost two hours to start after update to Pega Platform version 8.7.4

You update your Pega Platform deployment running Pega Customer Decision Hub™ version 7.3.1 to Pega Platform version 8.7.4 and notice that the application server takes almost two hours to start. Login to Dev Studio and navigation throughout the application are very slow, too. Each interaction in the application portal takes several minutes to complete.

Scenario 3 Application takes over an hour to start after update to a newer Pega Platform version 8 release

You update your Pega deployment from Pega Platform version 8.6.1 to Pega Platform version 8.7.3. The update runs smoothly with no issues encountered. However, application startup takes over an hour to complete.

Scenario 4 Server startup takes 75 minutes on Pega Platform version 8.4.6

After you update your Pega deployment to Pega Platform version 8.4.6, you notice that application server startup takes almost 75 minutes, and the application runs very slowly overall.

The logs show that it took the application approximately 63 minutes to start up. Reindexing on classes was not done, and old database statistics were being run during that 75-minute startup.

Scenario 5 Issue connecting Oracle 19C database from Pega Platform version 8.2

Your Pega Platform version 8.2 deployment running on IBM WebSphere Application Server version 8.5.5.14 connected to an Oracle19c database throws an exception during Pega bootstrap.

Here are the environment configuration details:

Java Version: IBM Corporation 1.8.0_201

Configuration: wsjar:file:/local/web/WebSphere855/AppServer/profiles/asp1/installedApps/vsin30u8215CellPEGA-QA/prpc_j2ee14_ws.ear/APP-INF/lib/prresources.jar!/prbootstrap.properties

Pega-EngineCode:

Database: Oracle Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Driver: Oracle JDBC driver 12.1.0.1.0

You see the following errors:

(!) Unable to use given database configuration

(!) Unable to locate classes on filesystem

Scenario 6 Slow restart of services after installing the hotfix to address the Log4j vulnerability

You installed hotfixes to address the Log4j vulnerability on Development, Pre-Production, and Production systems. Following standard operating procedure, your system administration team tried to restart the server after clearing the Pega Temp folder to ensure that the hotfix changes took effect. The server startup stalled and then was very slow, taking more than two hours to complete.

Scenario 7 Application server restart of Postgres RDS takes longer than expected

It typically takes your Tomcat application server and the Pega application five to ten (5-10) minutes to restart. But after you upgrade the Postgres Relational Database Service (RDS) from version 9.6.20 to version 11.10, Tomcat services and Pega initialization takes much longer, approximately 30 minutes.

Scenario 8 Pega Platform version 8.5.3 takes a long time to start up

Your Tomcat application server takes 166 minutes to start. The log displays error com.pega.pegarules.pub.clipboard.InvalidReferenceException: The reference .pyLabel is not valid. Reason: FUAInstance-NullMyStepPage and SQL times out with the message SQL State HY008.

Errors

Logs report various performance lags depending on the scenario.

Explanations

Environment - 3rd Party Issue

A defect or configuration issue in the operating environment: OJDBC driver conflicts

A defect or configuration issue in the operating environment: Database issue whereby fragmentation on startup-related tables like pr_engineclasses and pr_data_admin creates slow-running queries that cause server startup stall and very slow performance

Environment Issue – Database Statistics not updated

A defect or configuration issue whereby system startup loads all the classes from the pr_engineclasses table with out-of-date statistics that cause slowness

Out-of-date search indexes and database optimizer statistics lead to reduced startup performance after a Pega deployment update.

Environments

The problem was reported in the following environments and is likely to occur in other environments not listed here:

• Pega Platform version 8.2.2 on-premises
• Pega Platform version 8.3.1 on-premises
• Pega Platform version 8.4.3 on-premises
• Pega Platform version 8.4.6 on-premises
• Pega Platform version 8.5.3 on-premises
• Pega Platform version 8.6.5 on-premises
• Pega Platform version 8.7.3 on-premises
• Pega Platform version 8.7.4 on-premises

Solutions

Read the prerequisite documentation, follow best practices, and use or adapt the solutions for typical case scenarios to meet your needs.

Prerequisite
Best Practices
Scenario 1 Solution
Scenario 2 Solution
Scenario 3 Solution
Scenario 4 Solution
Scenario 5 Solution
Scenario 6 Solution
Scenario 7 Solution
Scenario 8 Solution

Prerequisite

See the Related Content, especially Pega Infinity Platform Support Guide for versions 8.1 - 8.8, all information pertaining to Java and JVMs, and Improving database performance.

Best Practices

Ensure that you adhere to standard operating procedure when performing Pega deployment activities, especially database administration, after updating to a major, minor, or patch release of the Pega Platform.  

• Update third-party software (Java, database management systems, and so on) to run the latest versions supported by Pega.
• Remove unused OJDBC drivers.
• Update database optimizer statistics and rebuild search indexes as a standard procedure after updating your Pega deployment.

Database tables, particularly system startup tables, require frequent defragmentation and index rebuilding along with re-computation of database statistics.

Scenario 1 Solution

Update third-party software. This is a common requirement after updating your Pega deployment.

For example, Scenario 1 required the following actions:

1. Update Java to ojdbc8.jar with version 19.3.0.0.
2. Update the Oracle 12c Database to the latest version, 12.2.0.1 or later. Version 12.1.0.1 is not recommended.
3. After completing Steps 1 and 2, have your database administrator (DBA) defragment the database indexes and tables as required. Your DBA must perform this step.
4. Update the statistics in the rules schema.

Reference: Pega Infinity Platform Support Guide for versions 8.1 - 8.8.

Scenario 2 Solution

Improve application startup time by running the following database commands on the Oracle database and the stored procedure PREPAY_RULES.SPPR_REBUILD_INDEXES(‘rules’).

BEGIN

   DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

END;

/

BEGIN

DBMS_STATS.GATHER_DICTIONARY_STATS;

END;

/

Reference: Updating statistics

Scenario 3 Solution

Update database optimizer statistics and rebuild search indexes as a standard procedure after updating your Pega deployment.

References: Updating statistics and Search indexes

Scenario 4 Solution

To ensure a smooth server and cluster startup after the updating your Pega environment, have your DBA perform the following steps:

1. Update database statistics and rebuild the database indexes, at minimum on the following tables:

pr_engineclasses

pr4_rule_vw

pr_index_reference

pr_assembled_classes

pr_sys_app_hierarchy_flat

pr_sys_app_ruleset_index

pr_sys_appcache_dep

pr_sys_appcache_entry

pr_sys_appcache_shortcut

pr_sys_cache_dep

pr_sys_cache_entry

pr_sys_cache_impl

pr4_rule_shortcut

pr4_rule_sysgen

2. Start the nodes in your environment, following this example of progressive startup in an environment with five (5) nodes.

1- Stream/Search/Background

1- Stream/Web

3 - Web nodes

1. Start the Stream/Search/Background.
2. Start the Stream/Web node.
3. Verify that these nodes are up and running.
4. Test the search functionality on these nodes and verify successful search results.
5. Start all Web nodes concurrently.
6. Verify that all Web nodes are up and running.

Startup time for all nodes is greatly improved.

References:  Updating statistics and Node management

Scenario 5 Solution 

Remove unused OJDBC drivers.

Reference: Creating and updating external database instances with JDBC URLs

Scenario 6 Solution 

To prevent performance lags caused by slow-running database queries on system startup tables, create a maintenance job to run every 15 days to defragment database tables, rebuild indexes, and re-compute database statistics.

After implementing this solution, you should see that system startup takes less than five minutes.

References: Defragmenting-tables, Database indexes,  Index tables, and Updating statistics.

Scenario 7 Solution

Update database statistics and re-compute them.

Reference: Updating statistics

Scenario 8 Solution

Performance guidance for production applications - Database

See also Improving database performance.

Related content

Pega Infinity Platform Support Guide for versions 8.1 - 8.8

Improving database performance

Defragmenting-tables

Updating statistics

Performance guidance for production applications - Database