Hello All,

we have this issue going on where we have been seeing this particular DNode exception, and when it happens we are seeing high CPU utilization, some nodes are reaching up to 300<. We had to restart the application every time. Can someone help us figure the cause. we also turned on logging level to INFO inside SMA for those particular couple of nodes. Below is the exception and some logs,

Exception:

.pega.dsm.dnode.impl.core.ExceptionWithInputRecord: com.pega.dsm.dnode.api.DNodeException: Cannot execute select statement<CR> ... 74 more<CR>Caused by: com.pega.dsm.dnode.api.DNodeException: Cannot execute select statement<CR> at com.pega.dsm.dnode.impl.cassandra.CassandraDao.executeStatement(CassandraDao.java:150)<CR> at com.pega.dsm.dnode.impl.cassandra.CassandraDataRepository.findBy(CassandraDataRepository.java:267)<CR> at com.pega.dsm.dnode.impl.dataset.cassandra.CassandraBrowseByKeysOperation$1.emit(CassandraBrowseByKeysOperation.java:56)<CR> at com.pega.dsm.dnode.impl

Logs:

100% java.lang.Thread.run():745
100% org.apache.tomcat.util.net.JIoEndpoint$Worker.run():926
100% org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(java.net.Socket):653
100% org.apache.coyote.http11.Http11Processor.process(java.net.Socket):856
100% org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, org.apache.coyote.Response):344
100% org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):102
100% org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):97
99.8% org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):145
99.8% org.jboss.as.web.security.SecurityContextAssociationValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):169
99.8% org.jboss.as.web.session.LockingValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):64
99.8% org.jboss.as.web.session.JvmRouteValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):92
99.8% org.jboss.as.web.session.ClusteredSessionValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):99
99.8% org.jboss.as.web.session.ClusteredSessionValve.handleRequest(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response, org.jboss.servlet.http.HttpEvent, boolean):134
99.8% org.apache.catalina.authenticator.AuthenticatorBase.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):420
99.8% org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):149
99.8% org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):231
99.8% org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse):214
99.8% org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse):295
99.8% javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse):847
99.65% javax.servlet.http.HttpServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse):754
99.65% com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse):121
99.65% com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(java.lang.Object[][], int, java.lang.Object, java.lang.Object[]):272
99.65% com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(java.lang.Object[][], int, java.lang.Object, java.lang.Object[]):223
99.65% com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(java.lang.Object[][], int, java.lang.Object, java.lang.Object[]):411
99.65% com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]):370
99.65% java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]):497
99.65% sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]):43
99.65% sun.reflect.GeneratedMethodAccessor71.invoke(java.lang.Object, java.lang.Object[]):unknown
99.65% com.pega.pegarules.web.impl.WebStandardImpl.doPost(com.pega.pegarules.internal.bootstrap.interfaces.IWebStandard, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse):374
99.65% com.pega.pegarules.web.impl.WebStandardImpl.makeEtierRequest(com.pega.pegarules.internal.bootstrap.interfaces.IWebStandard, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, java.util.Map, java.lang.Object, com.pega.pegarules.web.impl.DirectStreamWriter):574
99.65% com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngine(java.lang.String, java.lang.Object):223
99.65% com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngineInner(java.lang.String, java.lang.String, java.lang.Object):278
99.65% com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(java.lang.String, java.lang.Object[]):240
99.65% com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(java.lang.String, java.lang.Object[], java.util.Map):263
99.65% com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(java.lang.String, java.lang.Object[], java.util.Map, com.pega.pegarules.session.external.mgmt.EngineRunnable):315
82.59% com.pega.pegarules.integration.engine.internal.services.http.HTTPService.invoke(java.lang.Object[], java.util.Map):416
65.3% com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest():331
65.3% com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(java.lang.Object, int, long, java.lang.Object, java.lang.Object, java.lang.Object[], java.util.Map, java.lang.reflect.Method):841
65.3% com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(java.lang.Object, int, long, java.lang.Object, java.lang.Object, java.lang.Object[], java.util.Map, java.lang.Object, boolean, boolean, boolean, com.pega.pegarules.exec.external.util.TenantIdentifier):1008
65.3% com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(java.lang.Object, java.lang.Object, java.lang.Object[], java.util.Map, java.lang.Object, com.pega.pegarules.session.external.mgmt.IPRRequestor):1270
65.3% java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]):497
65.3% sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]):43
65.3% sun.reflect.GeneratedMethodAccessor72.invoke(java.lang.Object, java.lang.Object[]):unknown

More:

] (cassandra.net.MessagingService) INFO - Waiting for messaging service to quiesce

More:

Cassandra.service.GCIInspector WARN, Heap is full. You may need to reduce or memtable cache sizes.Cassandra will now flush up the two largest memtables. Adjust flush target memtables in yaml file at threshold if want to do this by cassandra automatically.

More: