Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 Santosh Nemani (SANTOSHN-1)
American Express

American Express
US
SANTOSHN-1 Member since 2013 2 posts
American Express
Posted: Nov 1, 2016
Last activity: Nov 10, 2016
Posted: 1 Nov 2016 14:21 EDT
Last activity: 10 Nov 2016 11:06 EST
Closed

Failed to process passivation queue after DB restart

We just did DB failover test while application is running, there was 2 minutes of outage and Database failover completed. We are able to login to Developer porta but in logs we are getting below errors

 

Recurring Error

2016-11-01 11:08:00,215 [.PRPCWorkManager : 1] [  STANDARD] [                    ] [     PegaRULES:07.10] (ternal.async.PassivationDaemon) ERROR   - Failed to process passivation queue:
java.lang.NoClassDefFoundError: com/pega/pegarules/session/internal/mgmt/base/PassivationMetadata

 

One Time Error

Caused by:
java.lang.RuntimeException: Problem during method invocation (invokeEngine)
        ... 48 more
Caused by:
java.lang.NoClassDefFoundError: com/pega/pegarules/session/internal/engineinterface/etier/engine/PRDiagnosticStream

***Updated by Moderator: Vidyaranjan| Updated Categories***

To see attachments, please log in.
Low-Code App Development
Java and Activities
Data Integration

Posted: 8 years ago
Posted: 2 Nov 2016 10:39 EDT
Santosh Nemani (SANTOSHN-1)
American Express

American Express
US

Here is the full stack trace

 

Show More

Here is the full stack trace

 

2016-11-01 10:43:00,147 [.PRPCWorkManager : 2] [  STANDARD] [                    ] [     PegaRULES:07.10] (ternal.async.PassivationDaemon) ERROR   - Failed to process passivation queue:
java.lang.NoClassDefFoundError: com/pega/pegarules/session/internal/mgmt/base/PassivationMetadata
        at com.pega.pegarules.session.internal.mgmt.base.FilesystemStorage.writeMetaData(FilesystemStorage.java:1036)
        at com.pega.pegarules.session.internal.mgmt.base.FilesystemStorage.seedMetadata(FilesystemStorage.java:1025)
        at com.pega.pegarules.session.internal.mgmt.base.FilesystemStorage.passivatePage(FilesystemStorage.java:429)
        at com.pega.pegarules.session.internal.mgmt.base.NodeRequestorMgt.savePage(NodeRequestorMgt.java:746)
        at com.pega.pegarules.session.internal.mgmt.base.NodeRequestorMgt.processPassivationQueue(NodeRequestorMgt.java:2433)
        at com.pega.pegarules.session.internal.async.PassivationDaemon.processPassivationQueue(PassivationDaemon.java:362)
        at com.pega.pegarules.session.internal.async.PassivationDaemon.run(PassivationDaemon.java:336)
        at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1177)
        at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:926)
        at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:777)
        at com.pega.pegarules.session.internal.async.PassivationDaemon.run(PassivationDaemon.java:298)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.AsyncServant.processAsyncTask(AsyncServant.java:267)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.AsyncServant.invoke(AsyncServant.java:166)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:315)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)
        at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:225)
        at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
        at java.lang.reflect.Method.invoke(Method.java:620)
        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:216)
        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:238)
        at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)
        at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.invokeEngine(Unknown Source)
        at com.pega.pegarules.session.internal.engineinterface.etier.mdb.PRAsync.processRequest(PRAsync.java:155)
        at com.pega.pegarules.session.internal.engineinterface.etier.asynchbeans.AsyncTask.run(AsyncTask.java:64)
        at com.pega.pegarules.session.internal.engineinterface.etier.asynchbeans.RunnableWrapper.run(RunnableWrapper.java:71)
        at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:271)
        at java.security.AccessController.doPrivileged(AccessController.java:360)
        at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:797)
        at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:222)
        at com.ibm.ws.asynchbeans.ABWorkItemImpl.run(ABWorkItemImpl.java:206)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)

 


Show Less
To see attachments, please log in.
Posted: 8 years ago
Posted: 2 Nov 2016 10:40 EDT
Santosh Nemani (SANTOSHN-1)
American Express

American Express
US

We are not explicitely mentioning which one, and from the stack trace it looks like file based

 

Show More

We are not explicitely mentioning which one, and from the stack trace it looks like file based

 

2016-11-01 10:43:00,147 [.PRPCWorkManager : 2] [  STANDARD] [                    ] [     PegaRULES:07.10] (ternal.async.PassivationDaemon) ERROR   - Failed to process passivation queue:
java.lang.NoClassDefFoundError: com/pega/pegarules/session/internal/mgmt/base/PassivationMetadata
        at com.pega.pegarules.session.internal.mgmt.base.FilesystemStorage.writeMetaData(FilesystemStorage.java:1036)
        at com.pega.pegarules.session.internal.mgmt.base.FilesystemStorage.seedMetadata(FilesystemStorage.java:1025)
        at com.pega.pegarules.session.internal.mgmt.base.FilesystemStorage.passivatePage(FilesystemStorage.java:429)
        at com.pega.pegarules.session.internal.mgmt.base.NodeRequestorMgt.savePage(NodeRequestorMgt.java:746)
        at com.pega.pegarules.session.internal.mgmt.base.NodeRequestorMgt.processPassivationQueue(NodeRequestorMgt.java:2433)
        at com.pega.pegarules.session.internal.async.PassivationDaemon.processPassivationQueue(PassivationDaemon.java:362)
        at com.pega.pegarules.session.internal.async.PassivationDaemon.run(PassivationDaemon.java:336)
        at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1177)
        at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:926)
        at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:777)
        at com.pega.pegarules.session.internal.async.PassivationDaemon.run(PassivationDaemon.java:298)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.AsyncServant.processAsyncTask(AsyncServant.java:267)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.AsyncServant.invoke(AsyncServant.java:166)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:315)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)
        at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:225)
        at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
        at java.lang.reflect.Method.invoke(Method.java:620)
        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:216)
        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:238)
        at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)
        at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.invokeEngine(Unknown Source)
        at com.pega.pegarules.session.internal.engineinterface.etier.mdb.PRAsync.processRequest(PRAsync.java:155)
        at com.pega.pegarules.session.internal.engineinterface.etier.asynchbeans.AsyncTask.run(AsyncTask.java:64)
        at com.pega.pegarules.session.internal.engineinterface.etier.asynchbeans.RunnableWrapper.run(RunnableWrapper.java:71)
        at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:271)
        at java.security.AccessController.doPrivileged(AccessController.java:360)
        at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:797)
        at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:222)
        at com.ibm.ws.asynchbeans.ABWorkItemImpl.run(ABWorkItemImpl.java:206)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)

 


Show Less
To see attachments, please log in.
Posted: 8 years ago
Updated: 8 years ago
Posted: 2 Nov 2016 7:33 EDT
Updated: 2 Nov 2016 7:32 EDT
Kevin Zheng (KevinZheng_GCS)
PEGA
Director, Technical Support
Pegasystems Inc.
US

Pega does not automatically recover for cases like database outage - at the moment (certainly a very desirable enhancement request). You would need to restart the application server to establish clean system. The errors you reported are typical of database outage scenario.

To see attachments, please log in.
Posted: 8 years ago
Posted: 2 Nov 2016 12:15 EDT
Ankur Das (@nkur.das_GCS) Principal Software Solutions Engineer
Pegasystems Inc.
US

Hi Santosh,

To add to Kevin's comments above. Its highly likely that due to the Database server outage or network issue between your Appserver and Database server, Websphere has raised a StaleConnectionException. Please review the logs (PegaRULES and SystemOut.log) to look for StaleConnectionException prior to the java.lang.NoClassDefFoundError. 

When the IBM Websphere Application Server’s Resource Adapter throws a StaleConnectionException, the connection(s) are evicted from the pool. Any application thread requesting a JDBC Connection from IBM Websphere’s JDBC pool when “Pre test” JDBC Connections in the pool feature is enabled (as in your case), Websphere will then try to pretest the JDBC Connection in the pool before handing it out to the requesting application thread. If you have reviwed and confirmed that a StaleConnectionException is being thrown the next question that comes to mind is - 

How do I deal with the StaleConnectionException in WebSphere Adapter for JDBC?

Please follow instructions under PDN and IBM Websphere Knowledgecenter articles below to deal with StaleConnections – 

1.    https://community.pega.com/support/support-articles/comibmwebspherececmstaleconnectionexception
2.    https://www.ibm.com/docs/en/was-nd/9.0.5?topic=SSAW57_9.0.5/com.ibm.websphere.nd.multiplatform.doc/ae/tdat_pretestconn.html

To see attachments, please log in.
Posted: 8 years ago
Posted: 3 Nov 2016 11:29 EDT
Santosh Nemani (SANTOSHN-1)
American Express

American Express
US

Thank you Kevin and Ankur. First of all we are doing DB failover testing, during failover DB was down for 2 minutes.So database was definately not available. But there is no stale connection error in database. We retried the failover without any change and this time there is no such exception in logs. So it looks intermittent. Do you think the connection pool settings need to be tweeked? Below is what my PegaRULES DB connection pool setting looks like.

                        ConnectionPool
                            -connectionTimeout 180
                            -maxConnections 100
                            -minConnections 50
                            -reapTime 180
                            -unusedTimeout 1800
                            -agedTimeout 360
                            -purgePolicy EntirePool
                            -testConnectionInterval 10
                            -testConnection true

 

To see attachments, please log in.
Posted: 8 years ago
Posted: 10 Nov 2016 11:06 EST
Lochana Durgada Vijayakumar (Lochan_DV)
PEGA
Senior Manager, Knowledge Management
Pegasystems Inc.
IN

Hello All,

Thank you for the great responses! Santosh has informed us, moderators, that he has not proceeded with further investigation since the issue has not recoccured. I will be closing this thread to future replies and will reopen the discussion once the investigation can be continued.

Regards,
Lochan  |  Community Moderator  |  Pegasystems Inc.

To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice