Hi, On our setup, I attempted to replace a failling Stream node by another one (with a different name). But I cannot start it.

Beforehand, we upgraded from 8.6.1 to 8.23.0

The cluster state during new stream node starting up :

2023-11-27 14:55:15,475 [e-registry-heartbeat] [  STANDARD] [                    ] [                    ] (.tasks.TopologyChangedDetector) INFO    - Services topology has changed. Current topology:                  DataFlow.BackgroundProcessing    naljbs700100.retraite.lan     Proprietary information hidden    NORMAL                          8.23.0    bc46f4ff-27cb-4e05-a3f3-0490f0fc3f79 This node ==>    DataFlow.BackgroundProcessing    naljbs700178.retraite.lan     Proprietary information hidden     JOINING           (new node)    8.23.0    5c4fcd69-3251-4c92-8b9b-7708cbc9baf6                  DataFlow.Search                  naljbs700100.retraite.lan     Proprietary information hidden    NORMAL                          8.23.0    6eef4091-24fe-4a33-a016-888196ebb7f8 This node ==>    DataFlow.Search                  naljbs700178.retraite.lan     Proprietary information hidden     JOINING           (new node)    8.23.0    17744249-8924-4eec-91e9-55dba45e2481                  StreamServer.Default             naljbs700100.retraite.lan     Proprietary information hidden    NORMAL                          8.23.0    b6d19976-c72e-486a-8d53-6b3d8fce7d10 This node ==>    StreamServer.Default             naljbs700178.retraite.lan     Proprietary information hidden     JOINING_FAILED                  8.23.0    178c0441-2382-41cb-a16f-ea67e9b5f261                  pega                             naljbm700033.retraite.lan     Proprietary information hidden    running                         8.23.0    f87da583-d5ed-4648-95fa-b62a9b9d0fa4                  pega                             naljbs700100.retraite.lan     Proprietary information hidden    running                         8.23.0    e7ea872f-0232-4b8c-a6a0-dfe68b1d7393 This node ==>    pega                             naljbs700178.retraite.lan     Proprietary information hidden     startingUp        (new node)    8.23.0    3d972cf9-e171-421b-91ef-8d4a5207323a

One error : 

2023-11-27 14:35:50,767 [StreamServer.Default] [  STANDARD] [                    ] [                    ] (prpc.service.ServiceDefinition) ERROR   - Failed to start service: [StreamServer.Default] com.pega.dsm.dnode.api.StreamServiceException: Cannot add new node. Make sure all listed nodes are available or reattach the volume from one of these nodes to this node.  [Node=naljbs700113.retraite.lan, broker=1002]         at com.pega.dsm.dnode.api.server.StreamServerService$StreamServiceStartOperation$2.emit(StreamServerService.java:675) ~[d-node.jar:?]         at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) ~[d-node.jar:?]

naljbs700113.retraite.lan is my old stream node.

I have not reference of this node in any of the following tables (I didn't drop them) : 

- pr_sys_statusnodes

- pr_sys_statusdetails

- pr_data_stream_nodes

- pr_data_stream_node_updates

- pr_data_stream_sessions

Another error :

Caused by: org.postgresql.util.PSQLException: ERROR: column "pxprocessingnodeid" does not exist   Position: 96         at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) ~[?:?]         at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) ~[?:?]         at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ~[?:?]         at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) ~[?:?]         at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) ~[?:?]         at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190) ~[?:?]         at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:177) ~[?:?]         at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:303) ~[?:?]         at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:442) ~[?:?]         at com.pega.pegarules.data.internal.store.DatabasePreparedStatementImpl.execute(DatabasePreparedStatementImpl.java:532) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.access.Lister.listWithResultPackager(Lister.java:421) ~[prprivate-data.jar:?]         ... 43 more

I don't know why this column is missing in the DB ?

A last error :

2023-11-27 14:37:26,416 [OBSCHEDULER_THREAD_5] [  STANDARD] [                    ] [                    ] (.SearchManager.ESSearchManager) WARN    - Checking cluster state failed. Reason: FTS is not initialized. com.pega.pegarules.priv.search.nextgen.FTSNotInitializedException: Initialization of full text search is either in progress or has failed on this node. Please check the logs if search initialization is complete.         at com.pega.platform.search.internal.ESSearchProviderEmbedded.getSearchNode(ESSearchProviderEmbedded.java:309) ~[search.jar:?]         at com.pega.platform.search.internal.ESSearchProviderEmbedded.getSearchNode(ESSearchProviderEmbedded.java:50) ~[search.jar:?]         at com.pega.platform.search.searchmanager.internal.ESSearchManagerEmbedded.getSearchNode(ESSearchManagerEmbedded.java:83) ~[search.jar:?]         at com.pega.platform.search.searchmanager.internal.ESSearchManagerEmbedded.getClusterState(ESSearchManagerEmbedded.java:270) ~[search.jar:?]         at com.pega.platform.search.searchmanager.internal.ESSearchManagerEmbedded.lambda$isReady$0(ESSearchManagerEmbedded.java:90) ~[search.jar:?]         at com.pega.platform.search.infrastructure.es.TimeoutHandler.withTimeoutHandling(TimeoutHandler.java:108) ~[search.jar:?]         at com.pega.platform.search.searchmanager.internal.ESSearchManagerEmbedded.isReady(ESSearchManagerEmbedded.java:89) ~[search.jar:?]         at com.pega.platform.search.searchmanager.internal.AbstractSearchManager.isReady(AbstractSearchManager.java:63) ~[search.jar:?]         at com.pega.platform.search.indexmanager.internal.ESIndexAdminEmbedded.verifyClusterIsReady(ESIndexAdminEmbedded.java:364) ~[search.jar:?]         at com.pega.platform.search.indexmanager.internal.ESIndexAdminEmbedded.getIndicesInfo(ESIndexAdminEmbedded.java:371) ~[search.jar:?]         at com.pega.platform.pegasearch.indexadmin.internal.PegaIndexAdminImpl.getIndicesInfo(PegaIndexAdminImpl.java:530) ~[pegasearch.jar:?]         at com.pega.pegarules.management.internal.system.operations.ClusterManagementUtils.searchIndexSettingsToObject(ClusterManagementUtils.java:221) ~[prprivate-management.jar:?]         at com.pega.pegarules.management.internal.system.operations.ClusterManagementAPI.getClusterSearchStateWithParams(ClusterManagementAPI.java:639) ~[prprivate-management.jar:?]         at com.pega.pegarules.management.internal.system.operations.ClusterManagementAPI.getClusterSearchStateForCurrentContext(ClusterManagementAPI.java:625) ~[prprivate-management.jar:?]         at com.pega.pegarules.management.internal.system.operations.ClusterManagementAPI.getSystemStateForSingleNode(ClusterManagementAPI.java:313) ~[prprivate-management.jar:?]         at com.pega.pegarules.management.internal.system.operations.ClusterManagementAPI.getSystemState(ClusterManagementAPI.java:244) ~[prprivate-management.jar:?]         at com.pega.pegarules.management.internal.system.operations.ClusterManagementAPI.persistNodeState(ClusterManagementAPI.java:1058) ~[prprivate-management.jar:?]         at com.pegarules.generated.activity.ra_action_pzpersistnodestate_95416bc413b38e19928c1cbb46614741.step1_circum0(ra_action_pzpersistnodestate_95416bc413b38e19928c1cbb46614741.java:165) ~[?:?]         at com.pegarules.generated.activity.ra_action_pzpersistnodestate_95416bc413b38e19928c1cbb46614741.perform(ra_action_pzpersistnodestate_95416bc413b38e19928c1cbb46614741.java:76) ~[?:?]         at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:2876) ~[prprivate-session.jar:?]         at com.pega.platform.executor.jobscheduler.internal.ActivityExecutor.runActivity(ActivityExecutor.java:59) ~[pega-executor.jar:?]         at com.pega.platform.executor.jobscheduler.internal.ActivityExecutor.executeActivity(ActivityExecutor.java:51) ~[pega-executor.jar:?]         at com.pega.platform.executor.jobscheduler.internal.ActivityProcessor.executeActivity(ActivityProcessor.java:73) ~[pega-executor.jar:?]         at com.pega.platform.executor.jobscheduler.internal.ActivityProcessor.execute(ActivityProcessor.java:59) ~[pega-executor.jar:?]         at com.pega.platform.executor.jobscheduler.internal.ActivityProcessor.run(ActivityProcessor.java:110) ~[pega-executor.jar:?]         at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1381) ~[prprivate-session.jar:?]         at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1124) ~[prprivate-session.jar:?]         at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1005) ~[prprivate-session.jar:?]         at com.pega.pegarules.session.internal.PRSessionProviderImplForModules.doWithRequestorLocked(PRSessionProviderImplForModules.java:83) ~[prprivate-session.jar:?]         at com.pega.platform.executor.jobscheduler.internal.ActivityProcessor.run(ActivityProcessor.java:101) ~[pega-executor.jar:?]

Can you help me ?