Stream Node stability in client managed cloud
Hi All,
We are using PEGA 8.8.1 on Client managed cloud and using auto heal ec2 instance for stream node (Stream and search both on same) and attached EBS volume to the EC2 instance.
Below are few issues that we are facing:
1. The stream node leave the cluster intermittently and we would need to rebuild the cloud ec2 instance for it to join the cluster
2. Whenever there is a need of new build i.e building a new cluster of nodes and old nodes are still there (including all nodes user, backgroound and stream node), the stream node has issue joining cluster. And as a workaround we kill all available nodes on that environment and clear enteries from pr_sys_statusnodes table and then build entire cluster again.
does anyone has encountered issues with Stream node joining cluster ? And if yes what did you do to resolve this issue permanently ?
Why only stream node has issue joining cluster ?
Does a failed stream node which did not join cluster can bring down other nodes?
Thanks
Eclatprime Digital Private Limited
SG
hi Amit,
we are using pega 882 and EKS cluster and we also face the same issue.
this issue is because of the PV and PVC. the best way to fix this is to do the below steps.
1. First uninstall the total deployment so that all web,batch, stream nodes will be down.
2. As you mentioned clear the pr sys nodes and below tables to clear the cache.
truncate table pegadata.pr_data_stream_nodes; truncate table pegadata.pr_data_stream_sessions; truncate table pegadata.pr_data_stream_node_updates; truncate table pegadata.PR_SYS_STATUSNODES;
3. Then delete all the PVC of the stream nodes and also make sure corresponding PV's are all cleared and retained.
4. Once u have done all these above steps now redeploy the application and now you can see it will create a new container and all pods will join the cluster without any issues.
Make sure you give atleast 1 core CPU for Stream nodes.
Macquarie
AU
Thank you for your input on this.
is my understanding correct that from approach above that it would be a longer down time for infra changes?
what are other approach which can reduce downtime for an environment?
Thanks
Eclatprime Digital Private Limited
SG
Based on my experience total task takes 20 min time and not sure how much critical your system is.
may be u can do this during your patching cycle as well