Pega 8.4 fresh install, Crazy amount of errors on catalina.out, eating RAM whats wrong?
Whats happening
Fresh install Pega 8.4(single Universal node, 2vms one postgres one tomcat), left untouched for couple of hours, afterwards it starts to go crazy on the catalina.out logs, also Postgres connections starts to drop, memory starts rising beyond Xmx, swap space is also used.
Have try several things in the connectors, DB, nothing seem to fix the issue
Whats happening
Fresh install Pega 8.4(single Universal node, 2vms one postgres one tomcat), left untouched for couple of hours, afterwards it starts to go crazy on the catalina.out logs, also Postgres connections starts to drop, memory starts rising beyond Xmx, swap space is also used.
Have try several things in the connectors, DB, nothing seem to fix the issue
| Errors Warning, Stacktraces Catalina.out | |
| * | 2020-04-09 23:09:55,117 [ow-Pulse-Executor:32] [ STANDARD] [ ] [ ] (ce.run.ThreadCompletionHandler) ERROR - Failed to finalize data flow thread for run [pyBIBatchIndexProcessor] |
| * | [ow-Pulse-Executor:32] [ STANDARD] [ ] [ ] (ce.run.ThreadCompletionHandler) ERROR - Failed to finalize data flow thread for run [pyBIIncrementalIndexer] java.lang.NoClassDefFoundError: com/pega/dsm/dnode/impl/dataflow/service/run/ThreadCompletionHandler$1 |
| * |
AsyncLogger error handling event seq=306053, value='Logger=com.pega.dsm.dnode.impl.dataflow.service.run.ThreadCompletionHandler Level=ERROR Message=Failed to finalize data flow thread for run [pyBIIncrementalIndexer]': java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/pattern/NotANumber 09-Apr-2020 23:09:55.121 WARNING [charlatan-watcher:7] org.apache.tomcat.jdbc.pool.PooledConnection.clearWarnings Unable to clear Warnings, connection will be closed org.postgresql.util.PSQLException: This connection has been closed. |
| * | [ow-Pulse-Executor:32] [ STANDARD] [ ] [ ] (.DataFlowServicePulseOperation) ERROR - Error while exec uting pulse task: ThreadFinalizationTask java.lang.reflect.UndeclaredThrowableException: null |
| * | Caused by: com.pega.pegarules.data.internal.rd.queryexec.ExecuteQueryException: An error occured on executing the query for the report definition - There was a problem getting a list: code: 0 SQLState: 08003 Message: This connection has been closed. DatabaseException caused by prior exception: org.postgresql.util.PSQLException: This connection has been closed. | SQL Code: 0 | SQL State: 08003 |
| * |
Caused by: com.pega.pegarules.pub.database.ConnectionException: There was a problem getting a list: code: 0 SQLState: 08003 Message: This connection has been closed. DatabaseException caused by prior exception: org.postgresql.util.PSQLException: This connection has been closed. | SQL Code: 0 | SQL State: 08003 at com.pega.pegarules.data.internal.access.ExceptionInformation.createAppropriateExceptionDueToDBFailure(ExceptionInformation.java:378) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.access.ExceptionInformation.createExceptionDueToDBFailure(ExceptionInformation.java:363) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.access.Lister.convertSqlExceptionToDatabaseException(Lister.java:607) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.access.Lister.listWithResultPackager(Lister.java:427) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.access.Lister.list(Lister.java:190) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:3006) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:2997) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.sqlapi.queryexec.NativeSQLQueryExecutor.executeQuery(NativeSQLQueryExecutor.java:98) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.sqlapi.queryexec.NativeSQLQueryExecutor.executeQuery(NativeSQLQueryExecutor.java:75) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.sqlapi.queryexec.NativeSQLQueryExecutor.executeQuery(NativeSQLQueryExecutor.java:51) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.executeQuery(NativeSQLListExecutor.java:130) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.executeSql(NativeSQLListExecutor.java:112) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.execute(NativeSQLListExecutor.java:82) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.execute(NativeSQLListExecutor.java:52) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.PRDataProviderImpl.executeNativeSQL(PRDataProviderImpl.java:1641) ~[prprivate-data.jar:?] at com.pega.pegarules.data.internal.PRDataProviderForModulesImpl.executeNativeSQL(PRDataProviderForModulesImpl.java:137) ~[prprivate-data.jar:?] at com.pega.platform.serviceregistry.internal.database.persistence.PlatformKeyValueStorePersistence.selectKey(PlatformKeyValueStorePersistence.java:147) ~[service-registry.jar:?] at com.pega.platform.serviceregistry.internal.database.persistence.PlatformKeyValueStorePersistence.get(PlatformKeyValueStorePersistence.java:88) ~[service- |
| * | WARNING [charlatan-watcher:7] org.apache.tomcat.jdbc.pool.PooledConnection.clearWarnings Unable to clear Warnings, connection will be closed. org.postgresql.util.PSQLException: This connection has been closed. |
SPECS:
Single node Pega 8.4 NodeType(Universal) on 2 GCP vms:
| Tomcat | Postgres | |
|---|---|---|
| Version | 9.0.33 | 11 |
| Specs | 2 CPU 14 RAM | 2CPU 7.5 RAM |
| postgresql.conf | max_connections = 150 shared_buffers = 1792MB effective_cache_size = 5376MB maintenance_work_mem = 448MB checkpoint_completion_target = 0.9 wal_buffers = 16MB default_statistics_target = 100 random_page_cost = 1.1 effective_io_concurrency = 200 work_mem = 6116kB min_wal_size = 1GB max_wal_size = 4GB max_worker_processes = 2 max_parallel_workers_per_gather = 1 max_parallel_workers = 2 max_parallel_maintenance_workers = 1 |
|---|
| Relevant JVM_OPTS |
-Xms4G -Xmx8G -XX:MaxMetaspaceSize=768m -Djava.security.egd=file:/dev/./urandom |
|---|---|
|
catalina.out |
about 2G and growing |
| contex.xml | <Resource name="jdbc/PegaRULES" auth="Container" type="javax.sql.DataSource" factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" driverClassName="org.postgresql.Driver" url="jdbc:postgresql://${db.ip}:5432/${db.name}" maxTotal="100" maxIdle="30" maxWaitMillis="10000" /> <Resource name="jdbc/AdminPegaRULES" auth="Container" type="javax.sql.DataSource" factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" driverClassName="org.postgresql.Driver" url="jdbc:postgresql://${db.ip}:5432/${db.name}" maxTotal="100" maxIdle="30" maxWaitMillis="10000" /> |
| server.xml | <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" /> |
Don't know what is wrong with this setup
Any ideas on how to fix this ? What would be the optimal parameters for a single node for a max of 5 users
like JVM options, server and context tomcat conf files
Postgres optimal defaults, etc.
Thanks
Pegasystems Inc.
US
There are several things I would suggest:
1. Add testonborrow=true in the context.xml (your context.xml uses tomcat connection pool, but yet the attributes are for dbcp2, check the latest installation guide. You can remove factory attribute, by default it would use dbcp2, your current settings would be alright.)
2. Never use Universal, it starts everything, the minimal node types are, Search,Stream,Backgroundprocessing,WebUser
3. Increase postgres max_connections to 1000.