Issue
Database deadlocks occur in the pr_operators table when applying customizations to the table’s activity logic. These deadlocks may also arise when using the model operator in an OAuth2 configuration. Changing the DSS to True eliminates the deadlocks but introduces a delay in search indexing, making the work items unsearchable.
Symptoms and Impact
The following symptoms occur as part of this issue:
-
Users are unable to access their production environments during updates and migration.
-
Deadlocks occur in the pr_operators table.
-
Requestors (operators who are logging) experience abnormal spikes.
-
High volume of inserts in the pr_operators table.
-
Nodes become unhealthy and go down.
-
Spikes in the CPU usage.
Root Cause
The database deadlocks occur in the pr_operators table as customizations to the table’s activity logic overrides the SkipDBsave DSS settings.
In some scenarios, database deadlocks may also occur when using the model operator in an OAuth2 configuration. When the CustomAuthentication activity authentication API is triggered, excessive queue entries are created in the pySASIncrementalIndexer queue processor, overwhelming the queue processor and causing system latencies.
Solution
A bug has been raised to set the DSS settings to True by default in the upcoming release. This change will resolve the database spikes in the pr_operators table when using the model operator in the OAuth2 configuration. This Known Issue document will be updated with release details when the fix for this issue is available.
As a best practice, avoid making customizations to the pr_operators table.