Kubernetes Inter POD Communication for Pega search
Pega PODS are unable to communicate with each other for Pega default Elastic Search.
Background:
We are trying to build Pega 8.5.1, in Client Cloud Azure with Docker Image and Kuberenets as container orchestrator. Pega is installed and first master node is brought up with node types as - BackgroundProcessing,Search,Stream,Batch. Search Index built properly and is in available staste in Search landing page.
However, when next node as WebUser node type is brought up Search index status is Unavailable in Landing page and Rule,Data search fails with error as-"Unable to reach master node" .
Observation:
Each POD has a Private IP and through Kubernetes Services has a Public/Cluster IP. In pr_sys_statusnodes table pyclusteraddress is occupied by PublicIP:5701 whereas pyindexaddress is occupied with privateIP:9300 for each node respectively.
Hazlecast is working properly after setting identification/cluster/public/address= Cluster IP of Master node in prconfig.xml and both node visible in Admin Studio-System & Nodes.
However when WebUser node is brought up/ restarted it throws below error in logs and search doesn't work.
Pega PODS are unable to communicate with each other for Pega default Elastic Search.
Background:
We are trying to build Pega 8.5.1, in Client Cloud Azure with Docker Image and Kuberenets as container orchestrator. Pega is installed and first master node is brought up with node types as - BackgroundProcessing,Search,Stream,Batch. Search Index built properly and is in available staste in Search landing page.
However, when next node as WebUser node type is brought up Search index status is Unavailable in Landing page and Rule,Data search fails with error as-"Unable to reach master node" .
Observation:
Each POD has a Private IP and through Kubernetes Services has a Public/Cluster IP. In pr_sys_statusnodes table pyclusteraddress is occupied by PublicIP:5701 whereas pyindexaddress is occupied with privateIP:9300 for each node respectively.
Hazlecast is working properly after setting identification/cluster/public/address= Cluster IP of Master node in prconfig.xml and both node visible in Admin Studio-System & Nodes.
However when WebUser node is brought up/ restarted it throws below error in logs and search doesn't work.
2021-01-14 12:27:18,282 [oundProcessControl-6] [ STANDARD] [ ] [ ] (BackgroundProcessEventListener) INFO - Starting service [Aggregation.Default] 2021-01-14 12:27:18,282 [oundProcessControl-6] [ STANDARD] [ ] [ ] (BackgroundProcessEventListener) INFO - Service [Aggregation.Default] started 2021-01-14 12:27:25,067 [il INITIALIZE_SEARCH] [ STANDARD] [ ] [ ] (SearchManager.ESSearchProvider) INFO - Successfully started ElastcSearch node 2021-01-14 12:27:55,140 [il INITIALIZE_SEARCH] [ STANDARD] [ ] [ ] (SearchManager.ESSearchProvider) WARN - Failed to connect to the master node. Please contact your system administrator. Reason: null 2021-01-14 12:27:55,140 [il INITIALIZE_SEARCH] [ STANDARD] [ ] [ ] (Manager.PegaSearchProviderImpl) INFO - FTS node initialization: updating node index directory to '' on host 'pegaforegroundnew-5d67dc8575-mbksg' - make sure that different nodes don't share it! 2021-01-14 12:28:25,357 [il INITIALIZE_SEARCH] [ STANDARD] [ ] [ ] (Manager.PegaSearchProviderImpl) ERROR - Cannot enable full-text search for node PEGAFOREGROUNDNEW-5D67DC8575-MBKSG. Possible cause: None of the search index host nodes are reachable. 2021-01-14 12:28:55,400 [il INITIALIZE_SEARCH] [ STANDARD] [ ] [ ] (artup.SearchInitializationTask) ERROR - Finished configuration of node for full-text search, with errors.
How to make PODS communicate and reach master node for Pega Search in Kubernetes?
Any help is highly appreciated.
Updated: 19 Jan 2021 7:59 EST
Pegasystems Inc.
AU
@Roshan Did you use Helm to deploy the pods or did that manually? Typically the pods are deployed within the same namespace and as such there is no network policy that is attached to the Pod, this allows the communications between the pods in the same namespace
Updated: 19 Jan 2021 7:59 EST
Tata Consultancy Services
GB
@dhang Yes Helm chart has been used to deploy the PODS. All PODS lie within same namespace. There is actually no issue with inter POD communication and hazelcast is working fine. However, when Pega search set up is done on subsequent node it fails with error as "Unable to reach master node". And search a rule also throws same error after system start up.
Updated: 19 Jan 2021 7:59 EST
Greenfield Technology Aktiengesellschaft
DE
@Roshan What about using an external ES container? Pega provides one at Docker Hub, it works for me. You need to register external ES either through a DSS or via prconfig.xml setting. See as well
(The root cause of your original problem seems to be that the search node name (peganode1 or whatever) is being resolved against the wrong IP address. Solution might be outside of Pega within network configuration.)
Updated: 19 Jan 2021 7:59 EST
Tata Consultancy Services
GB
We are yet to consider external ES container as the docker image provided has several security vulnerability. raised incident for the same.
Meanwhile trying the default Pega Elastic Search which infact works fine on master node but fails with slave nodes.