Applies to Pega Platform versions 7.1.6 through 8.3.1
Search node and cluster mismanaged
Cannot index custom properties
Search returns incomplete or empty results
Cannot add multiple Search nodes from the landing page
Indexing host nodes forming separate Elasticsearch clusters
When node is restarted, users cannot search for old work objects, rules, or data
NoClassDefFoundError and NoSuchMethodError exceptions
Search queries for Report Definitions malformed
Wrong search query generated for a value with spaces passed to a filter parameter
Unexpected items in search results for some values passed to pySearchString parameter
Missing items in search results cause partial values to be passed to pySearchString
Missing descendant work objects in search results
This article is related to other content about Pega Platform Search. See the required reading that is a prerequisite to using the information in this article:
Symptoms of Pega Platform Search issues can usually be traced to the following problems:
Search node and cluster mismanaged
Search queries for Report Definitions malformed
Understanding how these problems manifest themselves (symptoms and errors) and learning root cause analysis of the patterns (explanations) leads to solutions (sometimes a choice of suggested approaches).
Prerequisites
Make sure that you have completed the following prerequisites.
Stream service
The Stream service is required. Make sure that the Stream service is running on at least one node of the cluster. See the Related Content for articles about the Stream service.
Search node
A Search node type is required for running the queue processor FTSIncrementalIndexer. See the Related Content for articles about Queue Processors and Node Classification.
Required reading
Read the following articles before learning how to troubleshoot the additional issues for Search that are identified in this article. Required hotfixes are identified in the two Troubleshooting articles. The configuration procedure article pertains to Pega 8.1 and Elasticsearch client-server mode.
Troubleshooting Elasticsearch, Common Issues
Troubleshooting pyIndexDirectory null after nodes restart
Configuring client-server mode for Elasticsearch
Troubleshooting search and performance issues in Pega Infinity 8.x
Search node and cluster mismanaged
Recognize, understand, and solve issues related to the mismanagement of Search nodes in a clustered environment.
Cannot index custom properties
Search returns incomplete or empty results
Cannot add multiple Search nodes from the landing page
Indexing host nodes forming separate Elasticsearch clusters
When node is restarted, users cannot search for old work objects, rules, or data
Cannot index custom properties
Symptom
Business data properties are not indexed.
Explanation (Root Cause)
By default, Elasticsearch will index only the px, py, pz Pega properties and it will not index the business data.
Solution
- Create a data transform pySearchModel.
- Create or update instances of Data-CustomProperties-Search, which is driven by Data-Admin-System-Settings instance indexing/useDataInstance configuration.
If indexing/useDataInstance is set to true, Elasticsearch uses the Data-CustomProperties-Search rule; otherwise, it uses pySearchModel.
After indexing/useDataInstance has been set to true, even if all the data transforms have not been converted, the Elasticsearch index will look for instances of Data-CustomProperties-Search only.
You will also need to re-index if you add more or modify the existing Data-CustomProperties-Search instances in the the pySearchModel data transform.
Search returns incomplete or empty results
Solution
Ensure that the following conditions are met:
- Indexing is finished and available.
- The Dynamic System Setting indexing/distributed/search_enabled is set to true.
- On the Search landing page, the option Enable search indexing is selected.
- There are no broken items in FTSIncrementalIndexers or all records are scheduled. (Search runs through Queue Processors; therefore, the Stream service needs to be enabled. If you add a new Stream service, then you must re-index data manually.)
Cannot add multiple Search nodes from the landing page
Symptom
The new node is not added when the Save Settings action is submitted.
Explanation (Root Cause)
Index directory of the node is not specified.
Solution
- Add the following JVM argument to the Search node:
-Dindex.directory=/index_directory - Make sure that the temp folder is empty.
- Restart the JVM.
- Check the Search landing page: The node should be added automatically.
Indexing host nodes forming separate Elasticsearch clusters
Symptom
Search results differ between nodes.
Behavior
When all nodes are restarted at the same time, clustering misbehaves. The unexpected behavior is caused by the fact that none of the nodes join the specified Elasticsearch cluster. Instead, a random node outside of the designated Elasticsearch nodes elects itself as a master node and the remaining nodes cluster with that random node.
Explanation (Root Cause)
Delayed cluster formation
Solutions (Suggested Approaches)
- Choose a different database execution plan to speed up node startup and make the node discoverable.
- Disable the Min Master Node Auto-Detection setting and set the min_master_node count explicitly so that the current node waits for other nodes to join.
- Start the indexing host nodes first. Check the logs to see if the nodes are available.
When node is restarted, users cannot search for old work objects, rules, or data
Symptom
Data indexed before restart is not available for search.
Behavior
Data indexed before restart is not available, but new work can be searched.
Explanation (Root Cause)
The Node ID was changed after the nodes were restarted.
If the Node ID changed, a new directory is created in -Dindex.directory and reindexing starts by default. If there is already another indexing host node, it duplicates the index.
Solution
If reindexing does not start by default, explicitly initiate reindexing.
Indexing stuck in progress
Symptoms
Indexing status is incomplete for a long time. There are broken items in the Queue Processor and failure is caused by the following error:
Batch Indexing request failed. Possible cause: Quorum not met for writes to succeed. Action: Remove any offline nodes as index nodes.
Explanation (Root Cause)
The quorum is not met. The number of unreachable Search nodes exceeds a healthy limit.
See also Troubleshooting Elasticsearch, common issues, Best practices, Guideline 3.
Solution
Restart the unhealthy Search nodes.
Java class conflicts
Recognize, understand, and solve issues related to Java class conflicts.
NoClassDefFoundError and NoSuchMethodError exceptions
NoClassDefFoundError and NoSuchMethodError exceptions
Symptoms
Initialization of Elasticsearch fails with a message containing errors such as java.lang.NoClassDefFoundError.
Search is not working. Indices might be unavailable or corrupted. There are errors in the PegaRULES logs such as java.lang.NoSuchMethodError.
Explanation (Root Cause)
The Pega Platform relies on many internal and external JAR libraries. It might happen that some of the JAR libraries have been upgraded or changed, and new versions are not compatible with the rest of the Pega Platform.
Solution
- Use Log-PegaRULESMove to determine if there were some Import or Export changes in the Pega Platform that might have changed JAR libraries.
- Find problematic JAR libraries:
- Is there some information about the library in the logs?
- What JAR files have been changed by Import?
- Do you see the same libraries with multiple versions?
- Can all existing JAR files be found with the following query?
select * from pr_engineclasses;
- If you suspect that a JAR file is wrong (its version could be wrong), you can choose to take one of the following actions:
- Remove it or update it with the Import procedure in Pega Platform.
- Remove it from the table pr_engineclasses.
Search index lost
Recognize, understand, and solve issues related to a lost Search index.
No Search index backup
A lost Search index cannot be recovered if you have failed to back up the index.
Symptom
The Search index has been lost.
Explanation (Root Cause)
No backup of the Search index was created.
Solution: Create a backup of the Search index
If you are using a Pega Platform release prior to Pega 8.1.1, install the required hotfix. Then complete the steps below.
Required hotfixes are identified in Troubleshooting Elasticsearch, Common Issues and in Troubleshooting 'pyindexdirectory' null after nodes restart.
If you are using Pega 8.1.1 or a later release, complete the following steps.
Prerequisite: Before you start this procedure, ensure that the REST interface is enabled on Elasticsearch.
- On all Search nodes, create repository as the top-level directory and ensure that it is writable for the Operating System user who runs the Pega Platform application server.
- Create a new Dynamic System setting:
Short Description: indexing/distributed/path/repo
Owning Ruleset: Pega-SearchEngine
Setting Purpose: indexing/distributed/path/repo - Restart all Pega Platform Search nodes.
- Create a snapshot by following Elasticsearch Reference [7.3], Modules, Snapshot and Restore:
- Create a repository in Elasticsearch, for example
curl -d '{ "type": "fs", "settings": { "location": "/home/prpc/elastic-backup/fs" } }'
-H "Content-Type: application/json" -X POST http://$HOST:$PORT/_snapshot/my_backup?pretty
Ensure that the command returns the `acknowledged` message. - Issue a snapshot command, for example
curl -d '{ "indices": "rule" }' -H "Content-Type: application/json"
-X POST http://$HOST:$PORT/_snapshot/my_backup/$SNAPSHOT_NAME?pretty&wait_for_completion=true
- Create a repository in Elasticsearch, for example
- Restore by close index(es), perform restore, and open index(es):
# Close index ('rule' in this case).
curl -X POST http://$HOST:$PORT/rule/_close
# Restore
curl -X POST http://$HOST:$PORT/_snapshot/my_backup/$SNAPSHOT_NAME/_restore?pretty
# Re-open index
curl -X POST http://$HOST:$PORT/rule/_open
Search queries for Report Definitions malformed
Recognize, understand, and solve issues related to the incorrect use of pxRetrieveSearchData in Report Definitions.
Unexpected items in search results for some values passed to pySearchString parameter
Missing descendant work objects in search results
Wrong search query generated for a value with spaces passed to a filter parameter defined in a report definition
Symptom
Queries are created with the OR operator for values with spaces passed to a filter parameter in a report definition used by pxRetrieveSearchData.
Example: For the UK Postcode M1 5EA passed to the filter parameter pyCategory, the query is created with the following condition: (pyCategory:M1 OR pyCategory:5EA)
Explanation (Root Cause)
Each report definition defines its filters. Filters do not set the property pyUseTokenizer. It is equal to null for each filter. When pyUseTokenizer is null or true, then the filter value is analyzed by the Pega tokenizer. When value has more tokens (for example, M1 5EA has the tokens M1 and 5EA), then a query is created with the OR operator between the tokens (for example, pyCategory:M1 OR pyCategory:5EA).
Solution
Find the generated search query by enabling DEBUG as directed in Troubleshooting Elasticsearch, Common Issues, Best Practices, Guideline 9. The node on which the search request was made will show what the user was searching for, the generated Elasticsearch query, and the top results. It will also indicate if it was unable to reach the indexing host node. Developers are encouraged to use this approach to aid in determining the root cause of node-specific search problems.
Alternatively, follow these steps if you are using Pega 7.2.2 or a later release:
- Load a report definition into the clipboard.
- Set the property pyUseTokenizer to false for filtered items in the report definition.
- Use that virtual report definition in pxRetrieveSearchData by setting the property pyReportPageName.
A query is created with all the tokens in one string, for example, pyCategory:"M1 5EA" and default_operator=AND is applied.
Unexpected items in search results for some values passed to pySearchString parameter
Symptom
Unexpected items appear in search results for some values, for example, -Monday is passed to the pySearchString parameter.
Explanation (Root Cause)
Elasticsearch query string syntax is used for a value passed to pySearchString in pyRetrieveSearchData.
For example, the value -Monday is parsed by Elasticsearch query string syntax as 'Monday' must not be present.
Solution
Use pySearchString only when you need query string syntax features.
For more information, see Elasticsearch Reference [5.6] Query DSL, Full text queries, Query String Query, Query String Syntax.
Missing items in search results cause partial values to be passed to pySearchString or a filter parameter in report definition
Symptom
For example, items with hyphenated string some-thing are missing in search results when the value thing is passed to pySearchString in pyRetrieveSearchData or to a filter parameter in the report definition.
Explanation (Root Cause)
The hyphen character (-) is defined as a special character in the Pega custom tokenizer for Elasticsearch because it is used in item keys by Pega.
This customization allows searching by item keys (for example, search by Case ID).
Solution
- Use hyphens only for item keys
- Search using strings including all token characters defined in the Pega custom tokenizer.
The Pega custom tokenizer has the following token characters:- all letters
- all digits
- special characters: hyphen (-), underscore (_), exclamation point (!), percent (%), at sign (@) and period (.).
Missing descendant work objects in search results
Symptom
pxRetrieveSearchData does not return work objects from descendant classes of the class specified in pyReportClassName.
Explanation (Root Cause)
When a class in pyReportClassName is concrete and not a work pool, then the filter query will have a condition to get only work objects from that class. For example, consider the pxObjClass:App-Work-Example condition in the filter query for a class App-Work-Example.
Solution
The class should be concrete and a work pool. Then the filter query will have a condition to search all classes that start with that class name, for example, the pxObjClass:App-Work-Example* condition in the filter query.
Related content
Configuring client-server mode for Elasticsearch on Pega Platform
Troubleshooting the Stream service
Configuring the Stream service
Troubleshooting Elasticsearch issues in Pega 7.1.7 through 7.4