Question
Ministry of Education Saudi Arabia
SA
Last activity: 31 May 2022 5:20 EDT
Pega 8.7.1 ERROR - The setting alerts/streamUnderReplicatedPartitions is not present in PrConfigSettingsRegistry
Recently installed Pega 8.7.1 in Tomcat application servers with Windows, DB is SqlServer .
Application is running slow and as part of our observations we found lot of below Error entries in PegaRULES.log.
ERROR - The setting alerts/streamUnderReplicatedPartitions is not present in PrConfigSettingsRegistry. API is depreciated. Please use new API to get the setting information.
Where we can find PrConfigSettingsRegistry ? how to solve this issue?
Logs:
2022-05-30 12:58:33,414 [ver.Default-Pulse:12] [ STANDARD] [ ] [ ] ( basic.config.SettingsImpl) ERROR - The setting alerts/streamUnderReplicatedPartitions is not present in PrConfigSettingsRegistry. API is depreciated. Please use new API to get the setting information.
java.lang.Exception: Information : Setting does not exist in PrConfigSetting enum.
at com.pega.pegarules.exec.internal.basic.config.SettingsImpl.logMessageForDepreciatedAPI(SettingsImpl.java:803) ~[prprivate-exec.jar:?]
at com.pega.pegarules.exec.internal.basic.config.SettingsImpl.getAsInt(SettingsImpl.java:418) ~[prprivate-exec.jar:?]
at com.pega.decision.dsm.util.AlertUtils$KeyPerformanceIndicator$1.get(AlertUtils.java:347) ~[prpublic.jar:?]
Recently installed Pega 8.7.1 in Tomcat application servers with Windows, DB is SqlServer .
Application is running slow and as part of our observations we found lot of below Error entries in PegaRULES.log.
ERROR - The setting alerts/streamUnderReplicatedPartitions is not present in PrConfigSettingsRegistry. API is depreciated. Please use new API to get the setting information.
Where we can find PrConfigSettingsRegistry ? how to solve this issue?
Logs:
2022-05-30 12:58:33,414 [ver.Default-Pulse:12] [ STANDARD] [ ] [ ] ( basic.config.SettingsImpl) ERROR - The setting alerts/streamUnderReplicatedPartitions is not present in PrConfigSettingsRegistry. API is depreciated. Please use new API to get the setting information.
java.lang.Exception: Information : Setting does not exist in PrConfigSetting enum.
at com.pega.pegarules.exec.internal.basic.config.SettingsImpl.logMessageForDepreciatedAPI(SettingsImpl.java:803) ~[prprivate-exec.jar:?]
at com.pega.pegarules.exec.internal.basic.config.SettingsImpl.getAsInt(SettingsImpl.java:418) ~[prprivate-exec.jar:?]
at com.pega.decision.dsm.util.AlertUtils$KeyPerformanceIndicator$1.get(AlertUtils.java:347) ~[prpublic.jar:?]
at com.pega.decision.dsm.util.AlertUtils$KeyPerformanceIndicator$1.get(AlertUtils.java:344) ~[prpublic.jar:?]
at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:199) ~[guava-19.0.jar:?]
at com.pega.decision.dsm.util.AlertUtils$KeyPerformanceIndicator.getThreshold(AlertUtils.java:363) ~[prpublic.jar:?]
at com.pega.decision.dsm.util.AlertUtils.alertInternal(AlertUtils.java:90) ~[prpublic.jar:?]
at com.pega.decision.dsm.util.AlertUtils.alert(AlertUtils.java:85) ~[prpublic.jar:?]
at com.pega.decision.dsm.util.AlertUtils.alert(AlertUtils.java:69) ~[prpublic.jar:?]
at com.pega.dsm.dnode.api.server.StreamAlertService$UnderReplicatedPartitionLogger.alertForURP(StreamAlertService.java:167) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.server.StreamAlertService$UnderReplicatedPartitionLogger.alertIfNeeded(StreamAlertService.java:134) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.server.StreamAlertService.alertForOfflineAndUnderReplicatedPartitions(StreamAlertService.java:68) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.server.StreamServerService$StreamServicePulseOperation$1.emit(StreamServerService.java:963) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:117) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:106) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.health.ServiceHealthCheckRegistration$1.run(ServiceHealthCheckRegistration.java:122) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.health.ServiceHealthCheckRegistration$1.run(ServiceHealthCheckRegistration.java:113) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContextInternal(ServiceHelper.java:305) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContext(ServiceHelper.java:150) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.health.ServiceHealthCheckRegistration.lambda$registerPulseExecutor$0(ServiceHealthCheckRegistration.java:113) ~[d-node.jar:?]
at com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.run(MoreExecutors.java:656) ~[guava-19.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:59) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:56) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:124) ~[d-node.jar:?]
***Edited by Moderator Marije to add Support Case Details; update capability tags***
Pegasystems Inc.
GB
@KiranKS it sounds like the stream cluster is unhealthy.
What are Under replicated partitions (URPs)?
URPs are not a critical event in Kafka, but it requires attention. When partition is under replicated it is still possible to write to the partition and to read from the partition but partition isn't highly available.
Resolving Offline and URP Partitions
Offline or URP partitions event occurs when one or multiple Kafka brokers are not part of the Kafka cluster. Or one or multiple Kafka brokers aren't in a healthy state.
To resolve an Offline Partition, you must identify which Kafka broker(s) is causing offline partitions or URPs and fix it.
Check stream nodes state
Check healthy status of the stream nodes.
- In Dev Studio
- Go to the Stream Landing Page Configure -> Decisioning -> Infrastructure -> Services -> Stream
- Check that all the Stream nodes are in "NORMAL" state
There are nodes in 'UNREACHABLE' state
The node is offline. Bring back the node that is in 'UNREACHABLE' state
There are nodes in 'STOPPED' state
Identify the route cause. Stream Service can be in 'Stopped' state because the 1. Stream service was manually stopped from the Stream Landing Page or
@KiranKS it sounds like the stream cluster is unhealthy.
What are Under replicated partitions (URPs)?
URPs are not a critical event in Kafka, but it requires attention. When partition is under replicated it is still possible to write to the partition and to read from the partition but partition isn't highly available.
Resolving Offline and URP Partitions
Offline or URP partitions event occurs when one or multiple Kafka brokers are not part of the Kafka cluster. Or one or multiple Kafka brokers aren't in a healthy state.
To resolve an Offline Partition, you must identify which Kafka broker(s) is causing offline partitions or URPs and fix it.
Check stream nodes state
Check healthy status of the stream nodes.
- In Dev Studio
- Go to the Stream Landing Page Configure -> Decisioning -> Infrastructure -> Services -> Stream
- Check that all the Stream nodes are in "NORMAL" state
There are nodes in 'UNREACHABLE' state
The node is offline. Bring back the node that is in 'UNREACHABLE' state
There are nodes in 'STOPPED' state
Identify the route cause. Stream Service can be in 'Stopped' state because the 1. Stream service was manually stopped from the Stream Landing Page or
2. Kafka process isn't running
* Check Kafka logs <app-service-root>/kafka-1.1.0.x/logs/server.log on the node to identify the reason Kafka isn't running.
* Here are few scenarios why Kafka process isn't running:
* Kafka process was killed by the OOM Killer. In this case server.log ends abruptly, the latest logs don't contain shutdown related logs. Check OOM Killer log for more information.
* Kafka failed on Windows OS. See the main article Kafka Issues on Windows OS.
* Kafka halting because unclean leader election isn't allowed. Kafka initiates shutdown because it can't satisfy clean leader election. The example of such scenario is:
1. Two brokers of Kafka cluster are configured with unclean leader election = false.
2. Some messages are pushed to the Kafka, both brokers ISR
3. Stop broker 2. Broker 1 is now a leader for all its partitions
4. More messages are pushed to Kafka. Broker 1 is ahead of broker 2
5. Shutdwon broker 1.
if broker 2 comes online now, you will get offline partitions and writes will not work
If before broker 2 came online data on broker 1 was deleted/truncated, then when broker 1 comes online, broker 2 will start halting as unclean.leader.election is false.
3. Stream Service was stopped by the Service Registry because the Platform node is unhealthy. This is valid only for the Platform version 8.4+. Stream Service will be re-covered once the main issue is fixed.
All nodes are in 'NORMAL' state but there are URP or Offline partitions
This state means that Kafka process is running, but it's unhealthy. Identify the reason:
- Is number of broker partitions greater than 1000? See Partition count per broker.
Ministry of Education Saudi Arabia
SA
@MarijeSchillern Status of Stream nodes is NORMAL.
Updated: 31 May 2022 5:20 EDT
Pegasystems Inc.
GB
Do you have multiple nodes are configured as Stream nodes?
In How many nodes, Are you observing the reported issue?
Cause for Under Replicated partitions:
URPs occur when one of the kafka broker is lagging behind the other. In general under-replicated partitions isn't urgent issue and Kafka manages synchronization between the replicas automatically in background.
How to Identify and resolve the Issue:
1.Ensure that All the Kafka Brokers in the cluster is having active charlatan sessions
Connect to DB and execute below query:
a. select * from pegadata.pr_data_stream_nodes WHERE pymode != 'PERSISTENT' ;
b. select * from pegadata.pr_data_stream_sessions ;
2. Login to the Broker execute the below command on all the stream nodes. to check if there are any Under Replicated partitions:
./kafka-topics.sh --zookeeper localhost:2181 --describe --under-replicated-partitions
Our support team will need to know the output on problematic nodes to check the Under Replicated Partitions.
It might be worth taking the screenshots of the stream node details from the service landing page and collect the Pega Rules log, state-change log and controller log of stream nodes.
Do you have multiple nodes are configured as Stream nodes?
In How many nodes, Are you observing the reported issue?
Cause for Under Replicated partitions:
URPs occur when one of the kafka broker is lagging behind the other. In general under-replicated partitions isn't urgent issue and Kafka manages synchronization between the replicas automatically in background.
How to Identify and resolve the Issue:
1.Ensure that All the Kafka Brokers in the cluster is having active charlatan sessions
Connect to DB and execute below query:
a. select * from pegadata.pr_data_stream_nodes WHERE pymode != 'PERSISTENT' ;
b. select * from pegadata.pr_data_stream_sessions ;
2. Login to the Broker execute the below command on all the stream nodes. to check if there are any Under Replicated partitions:
./kafka-topics.sh --zookeeper localhost:2181 --describe --under-replicated-partitions
Our support team will need to know the output on problematic nodes to check the Under Replicated Partitions.
It might be worth taking the screenshots of the stream node details from the service landing page and collect the Pega Rules log, state-change log and controller log of stream nodes.
Please can you log a support incident, and provide the kafka server logs from the last restart in the support ticket?
Please provide the INC number here so that we can help track the issue.