Applies to Pega Platform™ versions 8.4.6, 8.5.6, 8.6.3, 8.6.4, 8.6.5, 8.7.x, 8.8.x, and 23.x
Symptoms
Explanation
Solution
Planned enhancement
Related content
Symptoms
You are using embedded search in your environment which is now deprecated in the latest releases, where Pega nodes are hosting Elasticsearch indexes and managing them and experience the following:
- Searches for newly created work fail.
- pyFTSIncrementalIndexer is not running or is turning off automatically.
Because these symptoms are caused by other conditions in your environment, you need to analyze the specifics of your environment to determine precisely what led to the reported behavior.
Here are some example scenarios.
Scenario 1: Index Status is RED or YELLOW
If the status of your indexes is RED, or in some cases YELLOW, the pyFTSIncrementalIndexer will automatically turn off.
How to gauge the health of Elasticsearch clusters and shards describes how to determine if this is the case.
Scenario 2: Stream Issue
If the Stream Service is not running in your environment or is having issues, this can cause the pyFTSIncrementalIndexer to not run. Navigate to the Queue Processor landing page in Admin Studio to check if your other Queue Processors are running and verify if there is a banner at the top of that page stating that the Stream Service is unavailable.
For more information on how to troubleshoot Stream service issues, refer to: Troubleshooting the Stream service
https://support.pega.com/question/error-stream-service-not-running-clus…
Scenario 3: Elasticsearch Failed to Initialize
With embedded search, all nodes must initialize Elasticsearch as part of startup. If Elasticsearch fails to initialize on an index host node, this can cause the pyFTSIncrementalIndexer to either stop on its own or in some rare cases cause it to report processing hundreds of thousands of records, but not drop the Ready to Process count over time.
You can identify if this has occurred by one of the following ways:
- Checking PDC to see if an OPS0010 has been reported, which will include the node it occurred on.
- Checking the pyIndexerState column in the data_schema.pr_sys_statusnodes table for a node reporting FAILED.
- Generating a current system state for your cluster and reviewing the SearchState.json file. This includes a section about the index host nodes and will also state whether the pyIndexerState has FAILED or not.
In these cases, we have typically relied on a restart of the affected index host node to recover, however we have created a package for clients still using embedded search on Pega versions 8.5.6+ that should allow your Pega application to self heal if this is encountered by restarting Elasticsearch on nodes where it failed to initialize as expected.
Solution:
The vast majority of issues we see with this can be resolved by downloading the ReinitFailedNodes.zip RAP file attached to this article and importing the file from Dev Studio using the default import options. The RAP file has both schema changes and rules that will need to be imported and applied.
This will add the Job Scheduler ReinitFailedNodes to your environment which will start running every hour on Search and BackgroundProcessing nodes and attempt to reinitialize Elasticsearch on those nodes if it has failed.
Given that Elasticsearch has already failed on a node where this will execute, this should have no negative impact to your environment. If this does not resolve the problem, look for errors during the last startup related to search not initializing as there may be something else going on that requires further investigation.
Related content
How to gauge the health of Elasticsearch clusters and shards
Using full-text search based in Elasticsearch
Manage batch indexing easily (Pega Platform 8.3)
