Elastic Search not working consistently
We recently upgraded to 7.2 from 7.1.8 and since then, elastic search not working consistently in development environment. We have 3 nodes and only one node is set for indexing on search landing page.
Also below DSS are set as suggested.
Pega-SearchEngine • indexing/distributed/search_enabled = "true"
We ran a automation script on search node to create 10 work objects and then search for each work object. Out of 10, search is able to retrieve 7 to 8 objects and 2 are failed. Even if I try to retrieve these objects manually latter, search did not fetch the results. And more surprisngly, on other nodes which are not set as search nodes, search is able to retrieve the results.
So, have few questions.
1. When we search on node which is not the search node, does it read the index files on search node and if so, how does it happen?
2. How to debug if certain work objects are not indexed? I don't find anything in the logs and appreciate if anyone provides the class for which I can enable debugging.
We recently upgraded to 7.2 from 7.1.8 and since then, elastic search not working consistently in development environment. We have 3 nodes and only one node is set for indexing on search landing page.
Also below DSS are set as suggested.
Pega-SearchEngine • indexing/distributed/search_enabled = "true"
We ran a automation script on search node to create 10 work objects and then search for each work object. Out of 10, search is able to retrieve 7 to 8 objects and 2 are failed. Even if I try to retrieve these objects manually latter, search did not fetch the results. And more surprisngly, on other nodes which are not set as search nodes, search is able to retrieve the results.
So, have few questions.
1. When we search on node which is not the search node, does it read the index files on search node and if so, how does it happen?
2. How to debug if certain work objects are not indexed? I don't find anything in the logs and appreciate if anyone provides the class for which I can enable debugging.
3. if all nodes are set as search nodes, how does prpc ensure that all index files are on sync across different nodes? Does PRPC generates index on one node and duplicates it to other nodes?
4.If ,multiple nodes are set as index nodes and when search is executed, does it retrieve only from first search node and other nodes are for fail over only?
Pegasystems
IN
1. When we search on node which is not the search node, does it read the index files on search node and if so, how does it happen?
It does read the files on the search node using the Elatic Search client API that is available in the library. It uses the port range 9300 to communicate. You can know the exact port being used for each node by looking at the pyIndexerAddress property of System-Status-Nodes instances.
>> 2. How to debug if certain work objects are not indexed? I don't find anything in the logs and appreciate if anyone provides the class for which I can enable debugging.
The first thing to check is whether FTSIncrementalIndexer agent is running on all nodes. If not, please enable it.
In the logs, search for any error messages. Note that you will need to check on the node where the work instance was created and also on the search node.
>> 3. if all nodes are set as search nodes, how does prpc ensure that all index files are on sync across different nodes? Does PRPC generates index on one node and duplicates it to other nodes?
Elastic Search internally takes care of replication of data across all nodes. This will ensure high availability and failover for search.
>> 4.If ,multiple nodes are set as index nodes and when search is executed, does it retrieve only from first search node and other nodes are for fail over only?
1. When we search on node which is not the search node, does it read the index files on search node and if so, how does it happen?
It does read the files on the search node using the Elatic Search client API that is available in the library. It uses the port range 9300 to communicate. You can know the exact port being used for each node by looking at the pyIndexerAddress property of System-Status-Nodes instances.
>> 2. How to debug if certain work objects are not indexed? I don't find anything in the logs and appreciate if anyone provides the class for which I can enable debugging.
The first thing to check is whether FTSIncrementalIndexer agent is running on all nodes. If not, please enable it.
In the logs, search for any error messages. Note that you will need to check on the node where the work instance was created and also on the search node.
>> 3. if all nodes are set as search nodes, how does prpc ensure that all index files are on sync across different nodes? Does PRPC generates index on one node and duplicates it to other nodes?
Elastic Search internally takes care of replication of data across all nodes. This will ensure high availability and failover for search.
>> 4.If ,multiple nodes are set as index nodes and when search is executed, does it retrieve only from first search node and other nodes are for fail over only?
Elastic Search will retrieve the data from any node so long as they are available. We don't set any parameters to the search query to look at the primary node first.
Cigna
US
Thanks for your response. But I can confirm that FTSIncrementalIndexer agent is running on all nodes and issue we are facing is little strange.
1. On search node, we created 10 work objects using script, but when we try to search one of the work object on this node, search did not retrieve any results.
2. On other nodes which are not set as search nodes, when we search for same work object, search is able to retrieve results. This is strange as this is supposed to retrieve results from search node and not sure how it finds record when we could not find it on search node itself
3. Verified the logs on search node and I don't see any exceptions related to indexing
Below is snap shot of pr_sys_status_nodes from our database where 1st one is search node..
| /eap_1/tempdir/PegaSearchIndex | Proprietary information hidden:9300 | Active |
| Proprietary information hidden:9300 | Active | |
| Proprietary information hidden:9300 | Active |
Appreciate any help or ideas to further debug this issue.
Pegasystems
IN
Please share a tracer output when search does not work.
Also, when you do run the tracer, please enable debug logging on com.pega.pegarules.search.internal.es
And, on the search landing page, what is the queue size shown in the agent section? Please share the screenshot.