Pega 8.5.1 upgrade - "StreamServiceException: Kafka data belongs to a node pega-stream-0 from the different cluster"
Hi,
1. Is the parameter, prconfig/cloud/isNodeAgnosticAdminStudio, only applicable to the web pods? Do you suggest that we use this parameter in our Openshift cluster running PEGA 8.5.1?
2. We faced issue with bring-up of stream pods after upgrade from 8.4.0 to 8.5.1 (on Openshift cluster). Do you have any suggestions on how to prevent the below exception from occurring during a production upgrade?
Workaround - After removing /opt/pega/kafkadata/prpc_node_id during stream pod startup, the pods are coming up. Would putting the 8.4 stream pods in Quiesce before scale-down, help to prevent this error from occurring after a in-place upgrade?
Cannot start service [StreamServer.Default]. Will retry in 180 seconds. Remaining attempts: 1][STACK][com.pega.dsm.dnode.api.StreamServiceException: Kafka data belongs to a node pega-stream-0 from the different cluster 631a648e-9867-48bd-a00d-1c8ffd9d35d1***Edited by Moderator: Pooja Gadige to add platform capability tag***
Pegasystems Inc.
IN
2. We faced issue with bring-up of stream pods after upgrade from 8.4.0 to 8.5.1 (on Openshift cluster). Do you have any suggestions on how to prevent the below exception from occurring during a production upgrade?
Would putting the 8.4 stream pods in Quiesce before scale-down, help to prevent this error from occurring after a in-place upgrade?
-- Yes. You can also decommission the stream nodes/pods before scale down to prevent this issue.
Cognizant
IN
As part of upgrade node id's were regenerated which interrupts KAFKA cluster communication ;
Your stream node registration will happen based on JVM arguments , but while boot strap process pega will create run time container for KAFKA using zookeeper and dynamically add node id while making JVM up for stream nodes ; you have pb with stale node id after post upgrade
1. Go to Decisioning Infrastructure Services Stream landing page and decommission all stream service instances
2. Shut down all JVM'S
3. Please truncate below tables
- truncate table pr_data_stream_nodes
- truncate table pr_data_stream_sessions
- truncate table pr_data_stream_node_updates
- truncate table PR_SYS_STATUSNODES
4. Restart the stream nodes first , then restart other nodes
Bhagath
Certified J2EE Architect / Pega CLSA
AI/ML expertise @ IIIT-H
Maximus
US
Hi Bagath,
With In-place or out-place upgrade,any actions need to be taken as part of Preupgrade and post upgrade activities from Kafka/Queueprocesser tasks.
What happens with the pending message processing when node ID regenerates, will it process pending messages after all node joins cluster.
Thanks
Deepa