Job scheduler execution issue
I noticed that in non production and production environments, intermittently the job scheduler execution stopped and in logs, following exception is getting displayed -
java.lang.IllegalStateException: Updating heartbeat took [47056]ms which is more than the heartbeat interval of [30000]ms at com.pega.platform.serviceregistry.internal.database.tasks.Heartbeat.execute(Heartbeat.java:130) ~[service-registry.jar:?] at com.pega.platform.serviceregistry.internal.database.tasks.TaskExecutor.execute(TaskExecutor.java:71) ~[service-registry.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_302] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_302] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_302] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_302] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_302] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_302]
------------------------------------------------------------------------
I noticed that in non production and production environments, intermittently the job scheduler execution stopped and in logs, following exception is getting displayed -
java.lang.IllegalStateException: Updating heartbeat took [47056]ms which is more than the heartbeat interval of [30000]ms at com.pega.platform.serviceregistry.internal.database.tasks.Heartbeat.execute(Heartbeat.java:130) ~[service-registry.jar:?] at com.pega.platform.serviceregistry.internal.database.tasks.TaskExecutor.execute(TaskExecutor.java:71) ~[service-registry.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_302] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_302] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_302] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_302] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_302] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_302]
------------------------------------------------------------------------
( JobScheduler) ERROR - Exceptio n in executing Job[pzDelayedQueueProcessorSchedule] java.lang.NullPointerException: null at com.pega.platform.executor.jobscheduler.util.ClusterUtilitiesImpl.getNodeIdFromUUID(ClusterUtilitiesImpl.java:74) ~[executor.jar:?] at com.pega.platform.executor.jobscheduler.scheduler.internal.ClusterJobDecorator.updateJobInfoForProcessing(ClusterJobDecorator.java:133) ~[executor .jar:?] at com.pega.platform.executor.jobscheduler.scheduler.internal.ClusterJobDecorator.shouldProceedWithCurrentRun(ClusterJobDecorator.java:69) ~[executor .jar:?] at com.pega.platform.executor.jobscheduler.scheduler.internal.ClusterJobDecorator.shouldExecute(ClusterJobDecorator.java:48) ~[executor.jar:?] at com.pega.platform.executor.jobscheduler.scheduler.internal.JobExecutionTemplate.executeJob(JobExecutionTemplate.java:44) ~[executor.jar:?] at com.pega.platform.executor.jobscheduler.scheduler.internal.JobExecutionTemplate.run(JobExecutionTemplate.java:35) ~[executor.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_302] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_302] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_302] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_302] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_302] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_302] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
This exception continues to be thrown for all job schedulers in particular node logs and on Admin studio that node is not displayed while on server management page node is up and running.
If anyone faced such issue , then please let me know.
LTIMindtree
SA
Hi,
As per the Error messages and Admin studio condition you posted, I can see that,
1. From infrastructure wise, Application Node types are not in Sync with Cluster.
2. it's the same cause why you're not seeing the node types in Admin Studio.
3. Environment needs maintenance to fix the issue. Once the Cleanup / Maintenance completed from the Infra side, Check on the Job Schedulers if the same issue still persist or not.