Discussion
Pegasystems Inc.
US
Last activity: 7 Jul 2025 10:09 EDT
Ask the Expert: Background Processing with Varun Nair & Sangeeth Bila
Welcome to our Ask the Expert session on Background Processing
Make sure to Favorite and Follow for updates via the Notification bell and/or weekly digest!
Session dates: 30 June - 4 July
Session opens to questions on 30 June!
Meet Your Experts:
I’m Varun Nair, a Senior Product Manager at Pegasystems with over two decades of experience in Information technology. I have utilized domain and technical knowledge to develop features such as Automations, Background Processing, and Event-driven application flows in Pega. I aim to offer a low code solution for developing complex enterprise-scale applications using Pega technologies.
I’m Sangeeth Bila and I'm a Senior Software Architect at Pega with over 15 years of experience in the software industry, including the past 10 years focused on the Pega platform. My work has spanned core areas such as session management, background processing, and the push notification framework. I've also contributed to key features involving operator presence and URL management.
Message from Your Experts:
Background Processing in Pega is provided by features like Queue Processors and Job Schedulers. These let you run long-running processes in the background without blocking user sessions and schedule resource-intensive tasks for low business hours. The features help implement application flow patterns like Async Processing, Massive Paralell Processing and Event driven flows.
Ask the Expert Rules
- Follow the Pega Support Center's Rules of Engagement
- This is not a Live Chat - Varun & Sangeeth will reply to your questions over the course of the session
- Questions should be clearly and succinctly expressed
- Questions should be of interest to many others in the audience
- Have fun!
Updated: 6 Jul 2025 17:13 EDT
CSC
US
Hi Varun & Sangeeth,
Our application(Pega 24.1.2) is running in Openshift Containers on Oracle DB. Most of our Queue Processor’s activity are running in context of “Background Processing” node type. Lately we noticed a lot of “java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached” error in PegaRULES log causing some pod restarts and queue processors going to "Not Running" state. But there was no related thread dumps or stuck threads in the logs. We have Elastic APM monitoring and we noticed that 4000+ threads has been used in one of the batch pods. We are checking if there was high volume processing happened during that time. But we have a question regarding the “Number of threads per node” on the Queue Processor rule instance , is it something application will hold that many threads idle even though only one message is being processed on that pod? Suppose say we have given Number of threads per node as 6 which means that when a message is consumed for processing by a Background Processing pod, 1 thread will be used and remaining 5 will be idle meaning not available for other processes in that batch pod? We are trying to find the root cause of this error from this angle but do let me know if any other possible reasons that we can analyze this from. Below is our batch tier resource config. Thanks! replicas: 4 cpuRequest: 1 memRequest: "12Gi" cpuLimit: 3 memLimit: "20Gi" initialHeap: "8000m" maxHeap: "12000m"
Pegasystems Inc.
IN
Let's take following 2 scenarios:
1) Number of threads per node = 3, number of BackgroundProcessing nodes = 2
3 threads will be created on each node. These threads will be created and consumes some resources even when they are not processing any messages.
2) Number of threads per node = 3, number of BackgroundProcessing nodes = 3
We can't have more threads than the number of partitions persent in Kafka topic. We have 6 partitions by default. So, only 2 threads will be created on each node and they consume resources.
Analyzing heap dumps for memory consumption by threads might help in debugging OutOfMemory error.
Updated: 7 Jul 2025 6:42 EDT
CSC
US
Thanks @bilas for your response. So this is my setup Number of threads per node = 6, number of BackgroundProcessing nodes = 4. So 6 threads will be created on each node. But only 1 or 2 will be used on a single node due to default partition 6. Remaining 4 threads will be idle consuming resources. So optimum # for "Number of threads per node" that i need to set will be 2 when we have BackgroundProcessing node count as 4. Please confirm if this understanding is correct. Thanks!
Updated: 7 Jul 2025 7:36 EDT
Pegasystems Inc.
IN
@Jey2705 No, we don't create idle threads.
If Number of threads per node =6 and background processing nodes are 4 then
- Two nodes will have 2 threads each( Only 2 will be created instead of 6).
- Other two nodes will have only one thread.
Idle threads will not be created.
CSC
US
Thanks for your response @bilas Sorry not idle thread. I meant threads created across all batch nodes which would be eventually consuming resources even when they are not processing any messages.
Pegasystems Inc.
IN
@Jey2705 - Starting Infinity'25 we have a new option for the QP - Automatically optimize for concurrency. Selecting this option would mean that platform will automatically manage the thread allocation and scale it synamically when required.
Updated: 7 Jul 2025 8:01 EDT
CSC
US
Thanks @Varun Nair for your response. That's great. Is that something we need to enable or Pega Platform will take care automatically? Also any reference link for the same to get some insights on the feature.
Pegasystems Inc.
IN
@Jey2705 - In Infinity' 25 , this has to be explicitly enabled at QP level through a new setting. I'm attaching a screenshot of the setting.
Updated: 6 Jul 2025 14:19 EDT
Capgemini
NO
Hi Experts, My question specifically concerns the functionality of Pega Constellation. Apologies if this topic is not relevant to this thread. :)
According to Pega documentation, the pySyncToConstellation Queue Processor is responsible for pushing customer static assets to the AppStatic server.
- Could you clarify what happens behind the scenes in the Java code of this QP activity when it executes?
- Why does this QP run on a Web Node instead of a Background Processing node?
- We are observing frequent broken items across all environments, with very limited information available in the item XML. What are the common causes of broken items in this QP?
Reference : https://docs.pega.com/bundle/platform/page/platform/background-processi…
Pegasystems Inc.
IN
@duraisankarI will check with the team owning the QP and provide the info.
Pegasystems Inc.
IN
@duraisankar pySyncToConstellation QP is used to sync Localization, Binary files, Components to Constellation AppStatic Service after performing rule resolution. 1. In the Java code, it calculates all impacted constellation applications and syncs them to the corresponding application static assets.
2. In the latest release, I noticed it is running on the Background Processing node. Could you please confirm the version of Infinity in which it is configured as a web node?
3. We can expect failures if it fails to connect to app static service. CC: @bilas
Updated: 7 Jul 2025 3:59 EDT
Capgemini
NO
@BharathKakaraparthi @bilas Thanks much for the prompt reply. I have following followup questions.
- Can you please ellabore what "Sync" means? Does Pega pushes these static files through REST from customer Pega instance to common Static Server which is shared across the customers in a region? It would great if you share some documents related to it.
- What are all the actions that may queue the items to this QP? (Just for example: Actions like save of Application Rule, Product import, etc)
- What are the common reasons for failures when attempting to connect to the appstatic service? Because requeuing the broken items didnt resolve the issue for us. We see exception "com.pega.platform.executor.queueprocessor.QueueProcessorException: Error opening instance <INSTANCE INSKEY>" Even though those rule instances exist in the system.
We are using Pega 24.2.2. I verfied from Admin Studio that it is running on BackgroundProcessing Node. I was referring to the Pega documentation where it is mentioned as WebUser node for this version. Ref : https://docs.pega.com/bundle/platform/page/platform/background-processi…
Pegasystems Inc.
IN
Could you please share the following details?
-
Are there any custom components in your application?
-
Are you using localization?
-
Are you encountering any runtime failures? In the broken queue item, which rule it failed to sync?
Capgemini
NO
-
Are there any custom components in your application? Yes, Pega Custom Constellation Components from Github
-
Are you using localization? Yes
-
Are you encountering any runtime failures? In the broken queue item, which rule it failed to sync? - Mostly Localization and Binary related files