Question
Intel
IN
Last activity: 3 Feb 2020 10:20 EST
Pega Service Restart Taking >30 minutes
Hi,
We are running Pega 8.1.3 OnPremise.
We are in the process of doing an upgrade of our environment, for which we are taking a backup of DB. And restoring the backup on a new database. The plan is to perform upgrade on this new DB.
But the problem we are facing is, pega is taking inordinately long time for services to start.
However, this is the the case with the original DB. The original DB takes 3-4 minutes for service start up. But the new backed up and restored DB is taking >30 minutes.
We are running Pega on Ubuntu with Postgresql as backend DB.
***Edited by Moderator Marissa to update platform capability tags****
Pegasystems Inc.
AU
Hi Kiran,
Thank you for posting your query. Could you please let us know from which version you are trying to upgrade? Also could you please share the server startup logs?
Intel
IN
Hi Uday,
Thank you for your response.
We are trying to upgrade to 8.1.6.
But , we have not reached this step yet. As a preliminary step before to upgrade we have just taken a backup of DB and restored it to a new DB, and trying to start pega on this new DB. And this is taking too long.
Services are however starting up with no errors, but its just that it's taking lot of time.
We are yet to start our upgrade activities.
I have attached here catalina and PegaRULES log file.
Thanks and Regards
Kiran
Proximus
BE
Hello,
The new infra is same as original one? same parameters/setup, memory, CPU, file system?
DB temp/undo, etc are same?
Have you started as pega/prpc system name instead of source DB system name?
Regards
Anthony
Intel
IN
Hi Anthony,
Thank you for your response.
We are not changing anything at the app layer. Same server, same memory, same CPU.
We are just changing the backed DB connection strings to point to new DB. The new DB is a backup and restored copy of the old one.
We are not creating a new system in Dev Studio.
Are you referring to Dev Studio configurations or something in context.xml file?
Thank and Regards
Kiran
Proximus
BE
ok, so from your app server which was connected to yourSourceDB, you just changed the datasource to be yourNewDB.
Is yourNewDB running on same server than yourSourceDB?
Can you please confirm that yourNewDB is having same TEMP/UNDO space than the Source one?
ok, so from your app server which was connected to yourSourceDB, you just changed the datasource to be yourNewDB.
Is yourNewDB running on same server than yourSourceDB?
Can you please confirm that yourNewDB is having same TEMP/UNDO space than the Source one?
You should make a compare of Source & New DB to be sure setup is same and that restore went fully fine.
I would also advice to put pega or prpc as system in your prconfig, as well as disabling agents to avoid any specific application processing.
Your could also make the non-recommended pega db cache clearing but since it's DEV and not working fine...(you could also go for clearing pegatempdir also)
Still, I would first focus on this newly created DB, its properties and its current load.
Regrds
Anthony
Intel
IN
Hi Anthony,
Yes, our backend database is running on the same server. Its just that we created a new DB. The connection string as in the sever name and port number are remaining the same.
We did compare the tables and record count on old and new DB. We are not seeing any difference in them.
We did a cache clear from Admin Studio, but running the API's. We did a cache clear for:
ABA cache
conclusion cache - both memory & database
declarative page cache
declarative rule cache
clear rule cache
We also cleared cache from DB server.
But still nothing has helped so far.
We kept a close eye on catalina.log file to observer the steps that are taking time.
There are few steps that have taken time. I have attached the catalina_excerpt.txt here.
Thanks and Regards
Kiran
Proximus
BE
During startup, have you checked whether there's no DB lock?
I already faced a strange (non explained) issue with a lock on table PR_SYS_APP_HIERARCHY_FLAT
I killed the lock and application went fine.
I've not found other way than killing this lock to have application working. this comes from time to time, not often.
You could have a look at activity on DB during startup. TEMP & UNDO tablespace are same between the 2 DBs, right? same as number of max sessions?
Regards
Anthony
Pegasystems Inc.
IN
Hi Kiran,
Did you verify size of all the cache tables like pr_sys_appcache_entry; pr_sys_appcache_dep; pr_sys_appcache_shortcut etc.
When engine starts it loads many cache like global declarative cache,conclusion cache etc.You should take backup of cache tables & truncate them.Restart the application & verify.
Please let me know If you need list of all the cache tables.
Thanks,
Abhinav
,