Question
US
Last activity: 7 May 2020 13:13 EDT
Rule Cache issue in multi-node environment
We are running Pega 7.4 Marketing - NBAA App on tomcat server with Oracle DB, have about 40 nodes working in production. We deploy changes to Decision tables and proposition data required by business on daily basis.
Sporadically, we are seeing issues with older version of decision tables getting picked up on few nodes in production although the latest version exists on that node.
We believe this is due to a cache Sync issue, verified system pulse is running on all nodes and we have shell unix scripts validating the pulse is latest on each node.
Can anyone provide inputs as to how we can deal this? Thankyou!!
Pegasystems Inc.
IN
Hi Shushruth,
Can you please share us the Pega RULES log from the the nodes where you are seeing cache issues ? Also, specify the problematic decision table instances names. This issue requires in depth analysis to find the root cause. You can also create incident with Pega GCS to troubleshoot this issue further.
Request you to check the below things:-
1.)
Seems to be VTable cache issue.
Can you check if page passivation is disabled with below setting (by any chance):
<env "initialization/persistrequestor/usepagelevelpassivation" value="false" />
Page passivation improves performance for production systems that may have 50 or more simultaneously connected requestors
https://community.pega.com/sites/default/files/help_v718/concepts/concepts2/conceptspassivation.htm
Also, Please check: https://community.pega.com/support/support-articles/decision-table-returns-old-value
2.)
In addition to above, try re-validating the application or application rulesets where you are experiencing the cache issues.
Please let us know if it helps.
Warm Regards,
Venkat
Pegasystems Inc.
IN
Hi,
Please try the following:
- Revalidate and save of the rulesets.
- Please verify that all the nodes are in sync and forming a cluster i.e. there is not any split-brain problem in the system.
- Try restarting the nodes one by one upon deployment this should help the nodes to join the same cluster.
- You can always reach out to Pega GCS for assistance.
US
Hello Venkat/Srivr -
This issue is sporadic and seen on Decision table updates migration to production. Decision table is setup with different Eligibility when rules for offer presentation. Old version of decision table getting picked up sometimes is the issue and sporadic.
Page passivation is neither disabled/enabled explicitly. Does it come enabled by default or should we have to declare explicitily for Vtable cache to work? How does the Vtable Cache work and how is it different from Rule Cache?
I observed in rule cache older versions of DT rule exists, is it valid because pulse agent should invalidate and remove the older versions, right? While in Vtable cache, DT rules doesnot exist.
FYI, we do not restart nodes for daily deployments of these updates, unless a code release happens which is every 3 weeks.
All nodes are in cluster and i do not see any hazlecast exceptions as such.
I did re-validate the Ruleset containing the WHEN rules but not the DT rules, this is because of the observation highlighted at this post: https://collaborate.pega.com/question/stringlist-column-comparisons%E2%80%94-instance-class-rule-obj-when-not-found
Let me know your thoughts.