Question
Transport for NSW
AU
Last activity: 30 Jan 2021 12:49 EST
Tomcat is not starting after upgrading from pega 722 to 83
We're doing an upgrade from Pega 722 to Pega 83 and after upgrade we found that tomcat is not up.
This is what we have done as part of upgrade.
1) Migrated 722 rules and data schema to new rules and data schema
2) Ran Pega83 upgrade.sh script on the migrated schemas and it went fine.
3) For app server setup, created new instance of tomcat and copied *.war files to webapps directory after giving db details in conf/context.xml file
4) Ran startup.sh - found tomcat connects to DB, does build rules cache etc and at the end of Catalina.out log found these errors and there is nothing logged after these logs and tomcat kind of hangs after this and is never up.
2020-02-13 14:11:12,092 [il INITIALIZE_SEARCH] [ STANDARD] [ ] [ ] (Manager.PegaSearchProviderImpl) ERROR - Cannot enable full-text search for node 344FCBA818ACA3ACC34290D33196669D. Possible cause : None of the search index host nodes are reachable. 2020-02-13 14:11:42,238 [il INITIALIZE_SEARCH] [ STANDARD] [ ] [ ] (artup.SearchInitializationTask) ERROR - Finished configuration of node for full-text search, with errors
Has anyone faced this or any ideas to solve this?
We're doing an upgrade from Pega 722 to Pega 83 and after upgrade we found that tomcat is not up.
This is what we have done as part of upgrade.
1) Migrated 722 rules and data schema to new rules and data schema
2) Ran Pega83 upgrade.sh script on the migrated schemas and it went fine.
3) For app server setup, created new instance of tomcat and copied *.war files to webapps directory after giving db details in conf/context.xml file
4) Ran startup.sh - found tomcat connects to DB, does build rules cache etc and at the end of Catalina.out log found these errors and there is nothing logged after these logs and tomcat kind of hangs after this and is never up.
2020-02-13 14:11:12,092 [il INITIALIZE_SEARCH] [ STANDARD] [ ] [ ] (Manager.PegaSearchProviderImpl) ERROR - Cannot enable full-text search for node 344FCBA818ACA3ACC34290D33196669D. Possible cause : None of the search index host nodes are reachable. 2020-02-13 14:11:42,238 [il INITIALIZE_SEARCH] [ STANDARD] [ ] [ ] (artup.SearchInitializationTask) ERROR - Finished configuration of node for full-text search, with errors
Has anyone faced this or any ideas to solve this?
***Moderator Edit-Vidyaranjan: Updated Platform Capability***
Pegasystems Inc.
US
Look into the below post discussions to understand and fix the search initialization issues post upgrade. The post pretty much covered all the points.
https://collaborate.pega.com/question/search-not-working-after-upgrade-83
Let me know if that helps!
Transport for NSW
AU
Hi Harish,
It progressed bit further from the earlier point after I changed a few jvm args but now it is stuck after the below lines. Do I need -pass the arg -DNodeType=Stream,Search,BackgroundProcessing,Webuser that you mentioned in the other post.
----
Hi Harish,
It progressed bit further from the earlier point after I changed a few jvm args but now it is stuck after the below lines. Do I need -pass the arg -DNodeType=Stream,Search,BackgroundProcessing,Webuser that you mentioned in the other post.
----
2020-02-13 17:27:25,822 [ervice-registry-task] [ STANDARD] [ ] [ ] (.tasks.TopologyChangedDetector) INFO - Services topology has changed. Current topology: This node ==> Aggregation.Default 344fcba818aca3acc34290d33196669d Proprietary information hidden NORMAL This node ==> DataFlow.BackgroundProcessing 344fcba818aca3acc34290d33196669d Proprietary information hidden NORMAL (was PENDING_JOINING) This node ==> DataFlow.Search 344fcba818aca3acc34290d33196669d Proprietary information hidden NORMAL (was PENDING_JOINING) This node ==> StreamServer.Default 344fcba818aca3acc34290d33196669d Proprietary information hidden JOINING This node ==> pega 344fcba818aca3acc34290d33196669d Proprietary information hidden startingUp
Pegasystems Inc.
US
Regarding the nodeType setting, it is best to configure it completely OR don't do it at all.
Without assigning any nodeType, Pega will anyway assign the 4 default nodeTypes required for Search, Agents, QueueProcessors to function.
Stream,Search,BackgroundProcessing,Webuser
The majority of the initialization is done and the final task is to initialize the stream search dataflow services. See if you can access the Designer Studio and check stream service landing page and make sure a node is added as a stream with NORMAL status.
If you see any issues with starting stream service on a node, look at this article to get an idea and ensure all are in place & not conflicting.
Transport for NSW
AU
Harish,
It is up now after shutting down another tomcat instance running on a different port on the same box. Not sure why one tomcat instance running on a different port altogether should have something to do with this instance which is running on another port?
Updated: 13 Feb 2020 21:25 EST
Pegasystems Inc.
US
The other tomcat instance is also running Pega application?
If yes, the ports used by the Apache Kafka for stream service on other tomcat instance might be conflicting and holding up another instance. The above article I mentioned have settings to configure unique ports for configuring stream services.
Let me know if that helps!
Transport for NSW
AU
Changed the port numbers in prconfig.xml as described in https://community.pega.com/knowledgebase/articles/advanced-configurations-stream-service. But no luck, it is still stuck at same point.
2020-02-14 15:23:26,842 [ervice-registry-task] [ STANDARD] [ ] [ ] (.tasks.TopologyChangedDetector) INFO - Services topology has changed. Current topology: This node ==> Aggregation.Default 344fcba818aca3acc34290d33196669d Proprietary information hidden NORMAL (new node) This node ==> DataFlow.BackgroundProcessing 344fcba818aca3acc34290d33196669d Proprietary information hidden NORMAL (new node) This node ==> DataFlow.Search 344fcba818aca3acc34290d33196669d Proprietary information hidden NORMAL (new node) This node ==> StreamServer.Default 344fcba818aca3acc34290d33196669d Proprietary information hidden JOINING (new node) This node ==> pega 344fcba818aca3acc34290d33196669d Proprietary information hidden startingUp
Pegasystems Inc.
US
Did you change all the three ports? I can't think of any other reason other than ports.
Retry after removing the kafka-1.1.xxx & kafka-data directories inside tomcat and restart the tomcat by ensuring you have all the unique ports.
It is also worth checking the controller.log inside kafka-1.1xxx/logs directory for any environmental issues.
Transport for NSW
AU
Yes, changed all the ports in prconfig.xml and tried. No luck.Even removed kafka directories with port numbers changed. Still it didn't work.
Only one of the instance works - which ever is started first.
Attached is a snippet from log. In the attached log, you will see that after the below log line, I don't see any activity from tomcat and kind of hangs or waiting for something. Then there is some error/exception in the log the next day that indicates that it is waiting for something.
This node ==> pega 344fcba818aca3acc34290d33196669d Proprietary information hidden startingUp
If there are any other known conflicting ports that are there between two pega tomcat instances that need to run on the same machine, please let me know.
Pegasystems Inc.
IN
Hi,
First start this server & then start other one,check if both starts or not.Please share error stack If you get any.
Thanks, Abhinav
Rulesware LLC
US
Hi Abhinav
My PE 8.5 is getting stuck at similar message when i am trying to restart after initial use.
I dont have multiple PE running in my laptop. Any idea on what could be the hung up?
Error: 2020-11-20 03:59:50,675 [e-registry-heartbeat] [ STANDARD] [ ] [ ] (.tasks.TopologyChangedDetector) INFO - Services topology has changed. Current topology: This node ==> StreamServer.Default ba801b385449c8aad6c640107293409a Proprietary information hidden JOINING (new node) 8.5.1 3125bf02-f704-44ed-b973-370dc6ededb1
after sometime it throws File not found exception ( but the file is present there)
Unable to access file:/C:/manoj85/PRPCPersonalEdition/tomcat/kafka- Proprietary information hidden/bin/file:C:/manoj85/PRPCPersonalEdition/tomcat/kafka- Proprietary information hidden/config/log4j2.xml java.io.FileNotFoundException: C:\manoj85\PRPCPersonalEdition\tomcat\kafka- Proprietary information hidden\bin\file:C:\manoj85\PRPCPersonalEdition\tomcat\kafka- Proprietary information hidden\config\log4j2.xml
Sinclair Broadcast
US
@MANOJPDN Was having the same issue (PE worked fine for first couple days, then wouldn't start the following weekend).
Deleting all files and folders in {install_dir}\tomcat\kafka-data\ fixed the issue on my side.