Question
Verizon Wireless
US
Last activity: 31 May 2017 23:20 EDT
Issues with the embedded Cassandra - DSM 722
Hi Team,
We are facing issues with the Decision data store in the Decisioning services wherein after adding the D-Node , the connection with embedded Cassandra is getting failed (STATUS - JOINING_FAILED) [Screenshot attached]
I have enabled the following logger com.pega.dsm.dnode.impl.cassandra.Cassandra and the below mentioned errors have been observed.
2017-05-26 08:47:02,097 [.PRPCWorkManager : 1] [ STANDARD] [ NBAHub:17.23] (rvice.operation.StartOperation) INFO rtdadmin - Starting service [DDS.Default] in client mode
2017-05-26 08:47:02,106 [.PRPCWorkManager : 1] [ STANDARD] [ NBAHub:17.23] (rvice.operation.StartOperation) INFO rtdadmin - Successfully started service [DDS.Default] in client mode
2017-05-26 08:47:09,009 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] (rvice.operation.StartOperation) INFO saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Starting service [DDS.Default]
2017-05-26 08:47:09,326 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) DEBUG saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Found java version [SemanticVersion{1.8.0}]. Will continue to start Cassandra
2017-05-26 08:47:09,348 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) INFO saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Cassandra instance is already unpacked. Skipping unpack
Hi Team,
We are facing issues with the Decision data store in the Decisioning services wherein after adding the D-Node , the connection with embedded Cassandra is getting failed (STATUS - JOINING_FAILED) [Screenshot attached]
I have enabled the following logger com.pega.dsm.dnode.impl.cassandra.Cassandra and the below mentioned errors have been observed.
2017-05-26 08:47:02,097 [.PRPCWorkManager : 1] [ STANDARD] [ NBAHub:17.23] (rvice.operation.StartOperation) INFO rtdadmin - Starting service [DDS.Default] in client mode
2017-05-26 08:47:02,106 [.PRPCWorkManager : 1] [ STANDARD] [ NBAHub:17.23] (rvice.operation.StartOperation) INFO rtdadmin - Successfully started service [DDS.Default] in client mode
2017-05-26 08:47:09,009 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] (rvice.operation.StartOperation) INFO saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Starting service [DDS.Default]
2017-05-26 08:47:09,326 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) DEBUG saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Found java version [SemanticVersion{1.8.0}]. Will continue to start Cassandra
2017-05-26 08:47:09,348 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) INFO saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Cassandra instance is already unpacked. Skipping unpack
2017-05-26 08:47:09,351 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) INFO saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Cassandra is not running. Will start it
2017-05-26 08:47:09,352 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) DEBUG saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Starting DNode bootstraping phase
2017-05-26 08:47:09,352 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) DEBUG saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Overwriting Cassandra yaml file
2017-05-26 08:47:09,357 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) DEBUG saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Overwriting Cassandra logback file
2017-05-26 08:47:09,358 [1df.cached.thread-15] [ STANDARD] [
2017-05-26 08:47:09,358 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) DEBUG saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Starting cassandra as a child process...
2017-05-26 08:47:09,369 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) INFO saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Started cassandra as a child process
2017-05-26 08:47:09,369 [dra-Output-Monitor-0] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) INFO saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Redirecting cassandra logs to PRPC
2017-05-26 08:47:10,388 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] ( impl.cassandra.Cassandra) INFO saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@xx- Waiting for Cassandra startup. Current state [NOT_RUNNING] CQL port listening [false]
2017-05-26 08:47:10,396 [1df.cached.thread-15] [ STANDARD] [ NBAHub:17.23] (rvice.operation.StartOperation) ERROR saswirtdzad01.sdc.xx.com|XX.XX.XX.XX ABC@vzw - Cannot start service [DDS.Default]
Regards,
Pradeep Pydi.
***Updated by Moderator: Marissa to mask proprietary information***
Verizon Wireless
US
pyErrorMessage DNodeException: Cannot bootstrap cassandra
pyErrorMessage DNodeException: Cannot bootstrap cassandra
pyErrorStackTrace com.pega.dsm.dnode.api.DNodeException: Cannot bootstrap cassandra at com.pega.dsm.dnode.impl.cassandra.Cassandra.bootstrap(Cassandra.java:213) at com.pega.dsm.dnode.api.DdsService.startCassandra(DdsService.java:688) at com.pega.dsm.dnode.api.DdsService.access$1200(DdsService.java:77) at com.pega.dsm.dnode.api.DdsService$DdsStartOperation$1$1.execute(DdsService.java:181) at com.pega.dsm.dnode.util.OperationWithLock$1.execute(OperationWithLock.java:54) at com.pega.dsm.dnode.util.OperationWithLock.doWithLock(OperationWithLock.java:89) at com.pega.dsm.dnode.util.OperationWithLock.doWithLock(OperationWithLock.java:58) at com.pega.dsm.dnode.util.OperationWithLock.doWithLock(OperationWithLock.java:39) at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeWithLockInternal(ServiceHelper.java:165) at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeWithLock(ServiceHelper.java:90) at com.pega.dsm.dnode.api.prpc.service.AbstractDsmService.executeWithLock(AbstractDsmService.java:416) at com.pega.dsm.dnode.api.DdsService.access$1500(DdsService.java:77) at com.pega.dsm.dnode.api.DdsService$DdsStartOperation$1.emit(DdsService.java:172) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:326) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:53) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:99) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:88) at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.doActualServerStart(StartOperation.java:112) at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.access$400(StartOperation.java:20) at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation$2.execute(StartOperation.java:93) at com.pega.dsm.dnode.util.OperationWithLock.doWithLock(OperationWithLock.java:89) at com.pega.dsm.dnode.util.OperationWithLock.doWithLock(OperationWithLock.java:58) at com.pega.dsm.dnode.util.OperationWithLock.doWithLock(OperationWithLock.java:39) at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeWithLockInternal(ServiceHelper.java:165) at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeWithLock(ServiceHelper.java:90) at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.initializeServerMode(StartOperation.java:90) at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.access$200(StartOperation.java:20) at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation$1.emit(StartOperation.java:61) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:326) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:53) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:99) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.toList(DataObservableImpl.java:78) at com.pega.dsm.dnode.impl.prpc.service.RemoteServiceOperation$ExecuteOperationMessage.call(RemoteServiceOperation.java:112) at com.pega.dsm.dnode.impl.prpc.service.RemoteServiceOperation$ExecuteOperationMessage.call(RemoteServiceOperation.java:72) at com.pega.dsm.dnode.api.prpc.service.CallableMessageWithContext$1.run(CallableMessageWithContext.java:33) at com.pega.dsm.dnode.api.prpc.service.CallableMessageWithContext$1.run(CallableMessageWithContext.java:30) at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:52) at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper$3.run(ServiceHelper.java:183) at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1286) at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1034) at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:843) at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContextInternal(ServiceHelper.java:176) at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContext(ServiceHelper.java:78) at com.pega.dsm.dnode.api.prpc.service.CallableMessageWithContext.call(CallableMessageWithContext.java:30) at com.pega.dsm.dnode.api.prpc.service.AbstractDsmService.handleMessageInternal(AbstractDsmService.java:230) at com.pega.dsm.dnode.impl.prpc.service.ServiceDeploymentImpl$PostMessageJobData.handleMessage(ServiceDeploymentImpl.java:382) at com.pega.dsm.dnode.impl.prpc.service.ServiceDeploymentImpl$PostMessageJob.execute(ServiceDeploymentImpl.java:352) at com.pega.dsm.dnode.impl.prpc.service.ServiceDeploymentImpl$PostMessageJob.execute(ServiceDeploymentImpl.java:349) at com.pega.pegarules.cluster.internal.PRPCTask.call(PRPCTask.java:110) at com.pega.pegarules.cluster.internal.PRPCTask.call(PRPCTask.java:52) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:187) at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) Caused by: com.pega.dsm.dnode.api.DNodeException: Unable to start DDS. Process has exited with code: 1 /log/temp/PegaTempDir_etr01_sbx/cassandra/bin/cassandra: test: unknown operator > at com.pega.dsm.dnode.impl.cassandra.Cassandra.awaitCassandra(Cassandra.java:439) at com.pega.dsm.dnode.impl.cassandra.Cassandra.startCassandraProcess(Cassandra.java:391) at com.pega.dsm.dnode.impl.cassandra.Cassandra.bootstrap(Cassandra.java:208) ... 57 more
Pegasystems Inc.
US
What is your platform, e.g., OS, is it AIX? If yes, this might be related to an issue we are investigating. What is the exact AIX version if yes?
Verizon Wireless
US
WebSphere Platform Proprietary information hidden [ND Proprietary information hidden cf101629.01] [IBMJAVA8 Proprietary information hidden cf101629.01] running with process name cell01\saswirtdzad01\srv01_etr01_sbx and process id 8
852
Host Operating System is SunOS, version 5.10
Java version = 1.6.0_115, Java Runtime Version = 1.6.0_115-b12 10_Jul_2016_10_03 solaris sparcv9 (SR8 FP25+IX90174), Java Compiler = HotSpot 64-Bit Tiered Compilers
, Java VM name = Java HotSpot(TM) 64-Bit Server VM
Pegasystems Inc.
US
To be sure, can you attach the SystemOut.log (including the error stack)? Are you running Pega platform 7.2.2? If yes, JDK 1.6 is no longer supported (which may not be related to your issue, but wondering why you would run an unsupported configuration).
Verizon Wireless
US
Right PEGA 722.
Thanks for the suggestion about JDK 1.6 (we will look about it).
Regarding the SystemOut.log,could you please provide additional details about what exactly your looking for ?
Pegasystems Inc.
US
Just want to inspect the complete SystemOut.log which has the error stack as shown from the landing page (as you already provided).