Symptoms

• Batch indexing operation fails with UnavailableShardsException or RemoteTransportException errors.
• The primary shard becomes inactive while the incremental indexing operation is stuck in the queue processor.
• FTS cluster state is RED.
• Rule search is often shut down in the development environment.

Error

ERROR - Failed to index dummy document for Index Type [rule]
com.pega.pegarules.priv.search.nextgen.FTSIndexRequestFailedException: Indexing request failed. Please contact your system administrator.
Caused by: org.elasticsearch.action.UnavailableShardsException: [rule][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[rule][0]] containing [delete {[rule][def][metainfoindex]}]]

Explanation

One of the Elasticsearch nodes is stuck in the initializing state. When the state of FTS cluster is RED, ElasticSearch is unable to process any query. The issue is due to a network failure or an exhausted disk space in the operating environment.

Environments

The problem was reported in the following environments:

• Pega Platform™ 7.3.1 on-premises
• Pega Platform 8.3.1 on-premises
• Pega Platform 8.5.3 in Pega Cloud® services 2.18.3
• Pega Platform 8.7.3 on-premises

Solution

Perform the following local change:

1. Run the pzResolveFTSIndexEntry activity against the search node.
2. Stop all Pega nodes.
3. Truncate the pr_sys_statusnodes table.
4. Delete existing indexes from search index folder or file directory.
5. For search nodes, the JVM argument is set to index.directory=/your/directory, to specify the path to the index directory. For non-search nodes, JVM argument should be empty.
6. Delete the Pega temp directory.
7. Restart all the Pega nodes.
8. Reindex data, work, and rules to make the status of data, work, and rules, as available.

Related content

Search and Reporting Service

How to Perform Elasticsearch Reindexing using the Command Line

pyFTSIncrementalIndexer not running or turning off automatically

Troubleshooting Search indexing failures