Stream Node Connection Issues
We are running our application in customer managed cloud. Few days ago, one of the the stream nodes went down and hence Auto scaling group tried to replace the same with a fresh instance.
As you know, during that time, other active nodes(web-user for example) tried to connect to the stream node(While it is coming up) that is starting up and it was in Pending-Joining status for a long time and then the logs indicated that the web-user node could be joined with the new stream node
StreamServer.Default stream-ip-100-83-227-225.ec2.internal Proprietary information hidden JOINING_FAILED (was PENDING_JOINING) 8.7.3 c3459de5-ec6d-472a-89c7-8f808fe8a7c0
During this window, the web user node experienced overheads, and api hits reaching this web -user node took more time(60 seconds).
After few minutes, the joining succeeded and the web-user node was able to join and it showed the new stream node as normal
Basically the web-user node was unstable for few minutes that caused hiccups in production.
Question :: I believe the stream node start up has to be quick in order to prevent this issue. What options shall be employed to expedite the stream node startup/joining?
or
Is it possible to defer the joining (from the webuser node) until the stream node is up and running to prevent overheads?
We are running our application in customer managed cloud. Few days ago, one of the the stream nodes went down and hence Auto scaling group tried to replace the same with a fresh instance.
As you know, during that time, other active nodes(web-user for example) tried to connect to the stream node(While it is coming up) that is starting up and it was in Pending-Joining status for a long time and then the logs indicated that the web-user node could be joined with the new stream node
StreamServer.Default stream-ip-100-83-227-225.ec2.internal Proprietary information hidden JOINING_FAILED (was PENDING_JOINING) 8.7.3 c3459de5-ec6d-472a-89c7-8f808fe8a7c0
During this window, the web user node experienced overheads, and api hits reaching this web -user node took more time(60 seconds).
After few minutes, the joining succeeded and the web-user node was able to join and it showed the new stream node as normal
Basically the web-user node was unstable for few minutes that caused hiccups in production.
Question :: I believe the stream node start up has to be quick in order to prevent this issue. What options shall be employed to expedite the stream node startup/joining?
or
Is it possible to defer the joining (from the webuser node) until the stream node is up and running to prevent overheads?
Any step/best practices to eliminate overheads will be helpful
LTIMindtree
SA
1. Once Application is brought down, you will need to truncate the below DB Tables.
- Truncate table <Data-Schema>.pr_sys_delayed_queue;
- Truncate table <Data-Schema>.pr_data_stream_nodes;
- Truncate table <Data-Schema>.pr_data_stream_sessions;
- Truncate table <Data-Schema>.pr_sys_statusnodes;
- Truncate table <Data-Schema>.pr_sys_statusdetails;
- Truncate table <Data-Schema>.pr_data_stream_node_updates;
- Truncate table <Data-Schema>.pr_perf_stats;
2. Removing the Kafka data from Stream node and restarting the node would resolve the issue.
Proximus
BE
Hello,
if you well have several streams nodes, then your other nodes should connect to the other (still up) stream node and not waiting for the one restarting. So your mentioned disturbance shouldn't happen.
About the speed of the start, you can check & adjust the value of below DSS
Pega-DecisionEngine • decision/services/startup/waitTimeIntervalSeconds
Regards
Anthony
Manheim
US
@Anthony_Gourtay That's the problem. The other node is up, however a background process running in pega detects a topology change(because the node crashed and the new one was brought up), it tries to connect to the newly spawned instance which is not quite ready yet(it takes few minutes for the stream node to come up).
About the waitTimeIntervalSeconds, is this the wait time the other nodes(say webuser trying to connect to a stream node) trying to connect to the stream node? In which case this has to be applied to the prconfig.xml of the webuser node?
LTIMindtree
SA
Question :: I believe the stream node start up has to be quick in order to prevent this issue. What options shall be employed to expedite the stream node startup/joining?
For your question, Please let me know whether you're using Actual On-Prem method or the containerization platform?