Applies to Pega Platform versions 7.2 through 8.3

Symptoms
Alternative solution
Related content
 

Symptoms

Avoid, prevent, or resolve common symptoms of Search indexing failure:

• Search Index file directory changes on restart
• Indexing fails with UnavailableShardsException
• Indexing fails with UnavailableShardsException for primary shard
• Search initialization error: Failed to obtain node locks

Symptom 1 Search Index file directory changes on restart

After you set the search index to a custom value and restart the application, you might see that the Search index file directory location is reset to the default location.

Symptom 1 Explanation

This problem can be caused by one or both of the following conditions:

• A defect or configuration issue in the operating environment
• Intentional cleanup of the run-time node configuration in the database

In Pega 7.2.2 and later releases, any nodes that are expected to permanently host indexes must have the index directory specified through a JVM argument.

Symptom 1 Solution

1. Add the following JVM setting:
   • For permanent index host nodes: -Dindex.directory=/your_index_directory
   • For nodes that are never expected to host indexes, add the JVM argument, but do not specify any directory (leave the value empty).
2. Make sure that the backward-compatibility Dynamic System Setting (DSS) indexing/explicitindexdir is not specified.
3. After making the changes, it is a best practice to restart PRPC nodes to include the changes.
   You can avoid the restart by specifying the same index directory path on the Search landing page.
   If you do this, verify that both settings (in the JVM argument and on the Search landing page) are identical.

Symptom 2 Indexing fails with UnavailableShardsException

When performing indexing from the Search landing page, you might see the UnavailableShardsException error.

Symptom 2 Error Message

2015-12-29 12:26:13,445 [PRPCWorkManager : 21] [  STANDARD] [     PegaRULES:07.10] (   internal.es.AbstractIndexer) WARN   Administrator-at-pega.com - Failed to index entry in batch: UnavailableShardsException[[data][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: com.pega.elasticsearch.action.bulk.BulkShardRequest@f87f010c]

Symptom 2 Explanation

A possible cause of reindexing issues is the FTS indexer being fooled into thinking that it is still indexing a different node than the one the user is currently running. 
This can happen when a database is moved from one environment to another or when a VM is cloned. 
This manifests as the indexing buttons on the Search landing pages being disabled, even though no indexing actions are happening on the node.

Symptom 2 Solution

1. Run the pzResolveFTSIndexEntry activity against the search node ID.
2. Delete the existing indexing folder in the search index file directory and create a new folder for search index directory.
3. Restart the FTS nodes one by one.
4. Perform the re-indexing.

Before restarting the nodes, make sure that the path to the index directory is permanently specified using the  JVM argument –Dindex.directory.
For more information, see Search Index file directory changes on restart.

Symptom 3 Indexing fails UnavailableShardsException for primary shard

When you run batch indexing or try to save anything, indexing fails with the error UnavailableShardsException: The primary shard is not active in the logs. It is not possible to index anything. Incremental indexing is stuck in the queue processors. FTS cluster state is RED.

Symptom 3 Error Message

ERROR - Failed to index dummy document for Index Type [rule]

com.pega.pegarules.priv.search.nextgen.FTSIndexRequestFailedException: Indexing request failed. Please contact your system administrator.

..............................................

Caused by: org.elasticsearch.action.UnavailableShardsException: [rule][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[rule][0]] containing [delete {[rule][def][metainfoindexnmlkjhgfdsazx]}]]

Symptom 3 Explanation

One of the Elasticsearch nodes is stuck in the INITIALIZING state. RED state means that the Elasticsearch cluster is not able to process any query. There are multiple reasons that can trigger this situation, typically related to environment issues that went unnoticed, for example, network failure or running out of disk space.

Symptom 3 Solution

1. Run the pzResolveFTSIndexEntry activity against the search node ID.
2. Delete the existing indexing folder in the search index file directory and create a new folder for search index directory.
3. Restart the FTS nodes one by one.
4. Perform the re-indexing of the data.

It is safe to empty the incremental queue processor prior to triggering the re-indexing.

Before restarting the nodes, make sure that the path to the index directory is permanently specified using the JVM argument –Dindex.directory.

For more information, see Search Index file directory changes on restart.

Symptom 4 Search initialization error: Failed to obtain node locks

Users report empty search landing pages and errors in the logs related to initialization of full-text search (FTS) functionality.

Symptom 4 Error Message

ERROR   - Failed to initialize full text search functionality for this node.

…

Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [index directory]  with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

Symptom 4 Explanation

This issue might happen on on-premise installations, where two PRPC nodes share the same file system. Multiple PRPC nodes are trying to use the same index directory, which is an illegal configuration.

Symptom 4 Solution

1. Make sure that the DSS indexing/explicitindexdir is not set,
2. Use the JVM argument -Dindex.directory  to specify index directories for search nodes, following the procedure described in Search Index file directory changes on restart. 
3. Make sure that each permanent index host PRPC node uses a distinct physical directory.
   If the PRPC nodes are running on the same physical machine, the paths must be different for each of them.

Alternative solution

This solution was offered in the SDoc Comment, 27July2023, by Aditya Bogga, Pega System Architect.

After performing the suggested Solutions above, if you still experience the problem, clean up the pegadata.pr_sys_statusnodes table and restart the server.

The pegadata.pr_sys_statusnodes table holds the status of the indexing. When indexes become corrupted, the status is set to Failed for the column pyindexerstate.

Related content

Pega Documentation

Using full-text search based in Elasticsearch

Pega Support Documentation

Troubleshooting pyIndexDirectory null after nodes restart 

Troubleshooting Pega Platform search

Troubleshooting indexing issues after update to Pega 7.3

Troubleshooting Search indexing failures

Troubleshooting Elasticsearch issues in Pega 7.1.7 through 7.4