Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 Naveen Saggam (NaveenS0683)
Telenet NV

Telenet NV
IN
NaveenS0683 Member since 2019 4 posts
Telenet NV
Posted: Jun 29, 2022
Last activity: Jun 30, 2022
Posted: 29 Jun 2022 19:03 EDT
Last activity: 30 Jun 2022 5:37 EDT
Closed

Inactive Sessions

Hi Team,

We are seeing lot of INACTIVE sessions on PEGA due to which nodes are going out of cluster.

We have around 100 connections per node and we have 6 nodes .

Could you please help how we can resovle inactive sessions 

Below is the error message we see in the logs.

---------------------------------------------

2022-06-28 11:11:54,631 [ow-Pulse-Executor:17] [  STANDARD] [                    ] [                    ] (        internal.access.Lister) ERROR   - There was a problem with the database when getting a list:

com.pega.pegarules.pub.database.ConnectionException: Database-General   Connection Exception    0               Cannot get a connection, pool error Timeout waiting for idle object

DatabaseException caused by prior exception: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object

| SQL Code: 0 | SQL State: null

        at com.pega.pegarules.data.internal.access.ExceptionInformation.createAppropriateExceptionDueToDBFailure(ExceptionInformation.java:379) ~[prprivate-data.jar:?]

Show More

Hi Team,

We are seeing lot of INACTIVE sessions on PEGA due to which nodes are going out of cluster.

We have around 100 connections per node and we have 6 nodes .

Could you please help how we can resovle inactive sessions 

Below is the error message we see in the logs.

---------------------------------------------

2022-06-28 11:11:54,631 [ow-Pulse-Executor:17] [  STANDARD] [                    ] [                    ] (        internal.access.Lister) ERROR   - There was a problem with the database when getting a list:

com.pega.pegarules.pub.database.ConnectionException: Database-General   Connection Exception    0               Cannot get a connection, pool error Timeout waiting for idle object

DatabaseException caused by prior exception: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object

| SQL Code: 0 | SQL State: null

        at com.pega.pegarules.data.internal.access.ExceptionInformation.createAppropriateExceptionDueToDBFailure(ExceptionInformation.java:379) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.access.ExceptionInformation.createExceptionDueToDBFailure(ExceptionInformation.java:364) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.store.ConnectionBrokerAdapter.getDataStoreConnection(ConnectionBrokerAdapter.java:264) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.store.ConnectionBrokerRouter.getDataStoreConnection(ConnectionBrokerRouter.java:71) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.store.ConnectionBrokerRouter.getDataStoreConnection(ConnectionBrokerRouter.java:65) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.store.AutoCommitTransaction.getDataStoreConnection(AutoCommitTransaction.java:316) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.store.AutoCommitTransaction.getConnection(AutoCommitTransaction.java:222) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.access.NativeSQLResultSetPackagerImpl.getDataStoreConnection(NativeSQLResultSetPackagerImpl.java:253) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.access.Lister.listWithResultPackager(Lister.java:373) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.access.Lister.list(Lister.java:190) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:3035) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:3026) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.sqlapi.queryexec.NativeSQLQueryExecutor.executeQuery(NativeSQLQueryExecutor.java:98) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.sqlapi.queryexec.NativeSQLQueryExecutor.executeQuery(NativeSQLQueryExecutor.java:75) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.sqlapi.queryexec.NativeSQLQueryExecutor.executeQuery(NativeSQLQueryExecutor.java:51) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.executeQuery(NativeSQLListExecutor.java:135) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.executeSql(NativeSQLListExecutor.java:117) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.execute(NativeSQLListExecutor.java:82) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.execute(NativeSQLListExecutor.java:52) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.PRDataProviderImpl.executeNativeSQL(PRDataProviderImpl.java:1678) ~[prprivate-data.jar:?]

        at com.pega.pegarules.data.internal.PRDataProviderForModulesImpl.executeNativeSQL(PRDataProviderForModulesImpl.java:138) ~[prprivate-data.jar:?]

        at com.pega.platform.serviceregistry.internal.database.persistence.PlatformKeyValueStorePersistence.selectRecursively(PlatformKeyValueStorePersistence.java:180) ~[service-registry.jar:?]

        at com.pega.platform.serviceregistry.internal.database.persistence.PlatformKeyValueStorePersistence.getAll(PlatformKeyValueStorePersistence.java:106) ~[service-registry.jar:?]

        at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) ~[?:?]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]

        at com.pega.platform.serviceregistry.internal.database.persistence.PrPersistenceProxy.invoke(PrPersistenceProxy.java:55) ~[service-registry.jar:?]

        at com.pega.platform.serviceregistry.internal.database.persistence.Proxies$PrKeyValueStorePersistence.invoke(Proxies.java:41) ~[service-registry.jar:?]

        at com.sun.proxy.$Proxy80.getAll(Unknown Source) ~[?:?]

        at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) ~[?:?]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]

        at com.pega.platform.serviceregistry.internal.database.persistence.ThrottlingPersistenceProxy.invoke(ThrottlingPersistenceProxy.java:45) ~[service-registry.jar:?]

        at com.pega.platform.serviceregistry.internal.database.persistence.Proxies$ThrottlingKeyValueStorePersistence.invoke(Proxies.java:55) ~[service-registry.jar:?]

        at com.sun.proxy.$Proxy80.getAll(Unknown Source) ~[?:?]

        at com.pega.platform.serviceregistry.internal.database.PlatformPersistentKeyValueStore.getAll(PlatformPersistentKeyValueStore.java:51) ~[service-registry.jar:?]

        at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.LeaderAndAuxiliaryElectionTask.execute(LeaderAndAuxiliaryElectionTask.java:67) ~[d-node.jar:?]

        at com.pega.dsm.dnode.impl.dataflow.service.DataFlowServicePulseOperation.handleTasks(DataFlowServicePulseOperation.java:107) ~[d-node.jar:?]

        at com.pega.dsm.dnode.impl.dataflow.service.DataFlowServicePulseOperation.access$000(DataFlowServicePulseOperation.java:47) ~[d-node.jar:?]

        at com.pega.dsm.dnode.impl.dataflow.service.DataFlowServicePulseOperation$1.run(DataFlowServicePulseOperation.java:88) ~[d-node.jar:?]

        at com.pega.dsm.dnode.impl.dataflow.service.DataFlowServicePulseOperation$1.run(DataFlowServicePulseOperation.java:82) ~[d-node.jar:?]

        at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?]

        at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContextInternal(ServiceHelper.java:301) ~[d-node.jar:?]

        at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContext(ServiceHelper.java:146) ~[d-node.jar:?]

        at com.pega.dsm.dnode.impl.dataflow.service.DataFlowServicePulseOperation.lambda$handlePulses$1(DataFlowServicePulseOperation.java:82) ~[d-node.jar:?]

        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) ~[guava-19.0.jar:?]

        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) ~[guava-19.0.jar:?]

        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) ~[guava-19.0.jar:?]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]

        at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:59) ~[d-node.jar:?]

        at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:56) ~[d-node.jar:?]

        at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?]

        at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:124) ~[d-node.jar:?]

Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object

        at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142) ~[tomcat-dbcp.jar:8.5.33]

        at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563) ~[tomcat-dbcp-9.0.37.jar:8.5.33]

        at com.pega.platform.datastore.connection.internal.jdbc.JdbcConnectionManager.getConnectionFromSource(JdbcConnectionManager.java:101) ~[datastore.jar:?]

        at com.pega.platform.datastore.connection.internal.jdbc.JdbcConnectionManager.get(JdbcConnectionManager.java:93) ~[datastore.jar:?]

        at com.pega.platform.logicaldatastore.connection.internal.ConnectionBrokerImpl.getConnection(ConnectionBrokerImpl.java:100) ~[logical-datastore.jar:?]

        at com.pega.pegarules.data.internal.store.ConnectionBrokerAdapter.getDataStoreConnection(ConnectionBrokerAdapter.java:245) ~[prprivate-data.jar:?]

        ... 51 more

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

        at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:431) ~[tomcat-dbcp.jar:8.5.33]

        at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345) ~[tomcat-dbcp.jar:8.5.33]

        at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) ~[tomcat-dbcp.jar:8.5.33]

        at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563) ~[tomcat-dbcp-9.0.37.jar:8.5.33]

        at com.pega.platform.datastore.connection.internal.jdbc.JdbcConnectionManager.getConnectionFromSource(JdbcConnectionManager.java:101) ~[datastore.jar:?]

        at com.pega.platform.datastore.connection.internal.jdbc.JdbcConnectionManager.get(JdbcConnectionManager.java:93) ~[datastore.jar:?]

        at com.pega.platform.logicaldatastore.connection.internal.ConnectionBrokerImpl.getConnection(ConnectionBrokerImpl.java:100) ~[logical-datastore.jar:?]

        at com.pega.pegarules.data.internal.store.ConnectionBrokerAdapter.getDataStoreConnection(ConnectionBrokerAdapter.java:245) ~[prprivate-data.jar:?]

        ... 51 more

 

Show Less

***Edited by Moderator Marije to add Capability tags***
To see attachments, please log in.
Pega Platform 8.5.3
Performance
System Administration
Data Integration
Installation and Deployment
Cloud Services
Communications and Media
System/Cloud Ops Administrator

Posted: 2 years ago
Posted: 30 Jun 2022 3:54 EDT
Mayank Jain (MayankJ2)
Deutsche Bank Operations International (DBOI)
Assistant Vice President
Deutsche Bank Operations International (DBOI)
IN

@NaveenS0683 - You mentioned that there are inactive sessions causing issues but your logs say a different story. Have you checked the connection pool setting? What are the values for maxTotal, maxIdle, maxWaitMillis, minIdle etc?

 

To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice