Question
Bradesco Seguros
BR
Last activity: 12 Mar 2025 20:06 EDT
Troubleshooting and avoiding delays in SLA Agent processing
Currently, I have the following scenario for SLA queue processing on the PegaSystems platform: Scheduled items - 515,325 in the queue - System-Queue-ServiceLevel, which currently has 7 background nodes to perform the processing. The ServiceLevelEvents agent is initiated every 30 seconds to process 500 records per execution. The average processing time has been 7 minutes. What should be analyzed and evaluated to improve this response time?
Accepted Solution
Updated: 12 Mar 2025 20:06 EDT
Pegasystems Inc.
GB
@GuiValino1984 Below aGENAI response from our bot:
Agent Configuration Mismatch: Modifying the ServiceLevelEvents Agent
Since the ServiceLevelEvents agent is native to the Pega-ProCom ruleset, here's how you can modify and specialize it:
@GuiValino1984 Below aGENAI response from our bot:
Agent Configuration Mismatch: Modifying the ServiceLevelEvents Agent
Since the ServiceLevelEvents agent is native to the Pega-ProCom ruleset, here's how you can modify and specialize it:
- Create a specialized version:
- Navigate to Records > SysAdmin > Agent > Rule-Agent-Queue
- Locate the ServiceLevelEvents agent in the Pega-ProCom ruleset
- Click "Save As" to create a specialized copy in your application ruleset
- This allows you to modify the agent without changing the base ruleset
- Configure mutual exclusion settings:
- In your specialized agent, go to the Advanced tab
- Select "Run this agent on only one node at a time" checkbox
- Select "Delay next run of agent across the cluster by specified time period"
- Configure an appropriate time window in "Block execution within this time window"
- This prevents overlapping executions across your cluster
- Adjust processing parameters:
- Increase the agent interval from 30 seconds to at least 7 minutes (matching your current processing time)
- Adjust the batch size (currently 500 records) based on your system capacity
- You can modify these in your specialized agent rule
- Promote the modified agent:
- Save your specialized agent to your application ruleset
- Disable the default ServiceLevelEvents agent
- Enable your specialized version
- Test thoroughly in a development environment before promoting to production
Resource Allocation and Scaling: Dedicated Nodes for SLA Processing
To create specific nodes dedicated to SLA processing:
- Node classification setup:
- In your Pega environment, classify specific application server nodes as "background processing nodes"
- This can be done through node classification configuration in your server settings
- For example, designate nodes with the property Pega-ProcessEngine/BackgroundProcessing=true
- Associate SLA processing with dedicated nodes:
- In your specialized ServiceLevelEvents agent, under the Advanced tab
- Configure "Node types" to specify which node types should run this agent
- Select only your designated background processing nodes
- This ensures SLA processing runs only on the dedicated nodes
- Modern approach: Queue processors instead of agents:
- For better scalability and performance, consider switching to queue processors:
- Enable the UseSLAQueueProcessor dynamic system setting (set to true)
- This switches processing from System-Queue-ServiceLevel to System-Message-QueueProcessor-DelayedItem
- Configure the pyProcessSLA queue processor to run on your dedicated background processing nodes
- Queue processors handle SLAs more efficiently than the traditional agent approach
- Load balancing across nodes:
- Ensure proper load distribution across your dedicated SLA processing nodes
- Monitor CPU, memory, and I/O utilization during peak processing times
- Adjust resources (scaling up) or add more nodes (scaling out) based on observed performance
These configurations will help resolve the mismatch between your agent execution frequency and processing time while ensuring SLAs are processed efficiently on dedicated resources. I recommend implementing these changes in a development environment first and monitoring the results before moving to production.
References:
- Pega-ProCom agents
- Avoiding race conditions for advanced agents
- Queue processing for SLAs
- Node classification
Pegasystems Inc.
GB
@GuiValino1984 ⚠ This is an answer from a GenAI-powered tool. All generated answers require validation against the provided references.
Here are several areas that should be analyzed and evaluated to improve response time:
1. Agent Configuration Mismatch
The most immediate issue appears to be a significant mismatch between your agent execution frequency (every 30 seconds) and the actual processing time (7 minutes). This means new agent executions are starting before previous ones complete, likely causing resource contention and performance degradation. Consider:
- Increasing the agent interval to match or exceed the processing time
- Implementing mutual exclusion settings to prevent overlapping executions
- Adjusting the processing batch size (currently 500 records) based on system capacity
2. Consider Queue Processor Instead of Agents
Pega recommends using queue processors instead of agents for SLA processing due to improved scalability, reliability, and performance:
@GuiValino1984 ⚠ This is an answer from a GenAI-powered tool. All generated answers require validation against the provided references.
Here are several areas that should be analyzed and evaluated to improve response time:
1. Agent Configuration Mismatch
The most immediate issue appears to be a significant mismatch between your agent execution frequency (every 30 seconds) and the actual processing time (7 minutes). This means new agent executions are starting before previous ones complete, likely causing resource contention and performance degradation. Consider:
- Increasing the agent interval to match or exceed the processing time
- Implementing mutual exclusion settings to prevent overlapping executions
- Adjusting the processing batch size (currently 500 records) based on system capacity
2. Consider Queue Processor Instead of Agents
Pega recommends using queue processors instead of agents for SLA processing due to improved scalability, reliability, and performance:
- Enable the
UseSLAQueueProcessordynamic system setting (set totrue) - This switches processing from System-Queue-ServiceLevel to System-Message-QueueProcessor-DelayedItem
- Queue processors are designed to evaluate a high number of items per second in a continuous data stream
3. Processing Activity Optimization
The underlying activities that handle each SLA item may need optimization:
- Review the processing logic for unnecessary database calls or complex operations
- Identify and optimize any activities with high CPU or memory usage
- Consider refactoring activities to use bulk operations where possible
- Look for opportunities to parallelize internal processing
4. Database Performance
Database performance often impacts SLA processing:
- Verify proper indexing on System-Queue-ServiceLevel and related tables
- Check for database contention or locking issues during peak processing times
- Consider database partitioning for large tables
- Analyze database query execution plans for queries related to SLA processing
5. Resource Allocation and Scaling
Your current setup with 7 background nodes may need adjustment:
- Evaluate CPU, memory, and I/O utilization during SLA processing
- Consider dedicating specific nodes exclusively to SLA processing
- For on-premises deployments, consider scaling up (more resources per node) or out (more nodes)
- Ensure proper load balancing across nodes
6. Monitoring and Metrics
Implement comprehensive monitoring to identify bottlenecks:
- Use Pega Diagnostic Center to analyze processing times and node utilization
- Monitor queue length trends over time to understand processing throughput
- Track SLA escalation events and correlate with processing delays
- Calculate effective throughput rates and compare with incoming workload
7. Performance Testing
Consider conducting controlled performance testing:
- Test different agent configurations (batch sizes, intervals)
- Compare agent-based processing with queue processor-based processing
- Measure the impact of different resource allocations
References:
Bradesco Seguros
BR
Thank you for the feedback. I have some questions. We are still working on version 8.6.6 - Regarding the items Agent Configuration Mismatch and Resource Allocation and Scaling.
Agent Configuration Mismatch:
- Increase the agent interval to match or exceed the processing time.
- Implement mutual exclusion settings to prevent overlapping executions.
- Adjust the processing batch size (currently 500 records) based on system capacity.
Since the agent is native to the ServiceLevelEvents platform, which is in the Pega-ProCom ruleset, how could I promote this modification? Can I specialize this agent? Can you give me more details?
Resource Allocation and Scaling:
- Consider dedicating specific nodes exclusively to SLA processing.
- For on-premises deployments, consider scaling up (more resources per node) or out (more nodes).
Can I create specific nodes for SLA and indicate the agent to use these resources for processing? Given the same scenario, which is an OOTB rule, how can I apply this?
Accepted Solution
Updated: 12 Mar 2025 20:06 EDT
Pegasystems Inc.
GB
@GuiValino1984 Below aGENAI response from our bot:
Agent Configuration Mismatch: Modifying the ServiceLevelEvents Agent
Since the ServiceLevelEvents agent is native to the Pega-ProCom ruleset, here's how you can modify and specialize it:
@GuiValino1984 Below aGENAI response from our bot:
Agent Configuration Mismatch: Modifying the ServiceLevelEvents Agent
Since the ServiceLevelEvents agent is native to the Pega-ProCom ruleset, here's how you can modify and specialize it:
- Create a specialized version:
- Navigate to Records > SysAdmin > Agent > Rule-Agent-Queue
- Locate the ServiceLevelEvents agent in the Pega-ProCom ruleset
- Click "Save As" to create a specialized copy in your application ruleset
- This allows you to modify the agent without changing the base ruleset
- Configure mutual exclusion settings:
- In your specialized agent, go to the Advanced tab
- Select "Run this agent on only one node at a time" checkbox
- Select "Delay next run of agent across the cluster by specified time period"
- Configure an appropriate time window in "Block execution within this time window"
- This prevents overlapping executions across your cluster
- Adjust processing parameters:
- Increase the agent interval from 30 seconds to at least 7 minutes (matching your current processing time)
- Adjust the batch size (currently 500 records) based on your system capacity
- You can modify these in your specialized agent rule
- Promote the modified agent:
- Save your specialized agent to your application ruleset
- Disable the default ServiceLevelEvents agent
- Enable your specialized version
- Test thoroughly in a development environment before promoting to production
Resource Allocation and Scaling: Dedicated Nodes for SLA Processing
To create specific nodes dedicated to SLA processing:
- Node classification setup:
- In your Pega environment, classify specific application server nodes as "background processing nodes"
- This can be done through node classification configuration in your server settings
- For example, designate nodes with the property Pega-ProcessEngine/BackgroundProcessing=true
- Associate SLA processing with dedicated nodes:
- In your specialized ServiceLevelEvents agent, under the Advanced tab
- Configure "Node types" to specify which node types should run this agent
- Select only your designated background processing nodes
- This ensures SLA processing runs only on the dedicated nodes
- Modern approach: Queue processors instead of agents:
- For better scalability and performance, consider switching to queue processors:
- Enable the UseSLAQueueProcessor dynamic system setting (set to true)
- This switches processing from System-Queue-ServiceLevel to System-Message-QueueProcessor-DelayedItem
- Configure the pyProcessSLA queue processor to run on your dedicated background processing nodes
- Queue processors handle SLAs more efficiently than the traditional agent approach
- Load balancing across nodes:
- Ensure proper load distribution across your dedicated SLA processing nodes
- Monitor CPU, memory, and I/O utilization during peak processing times
- Adjust resources (scaling up) or add more nodes (scaling out) based on observed performance
These configurations will help resolve the mismatch between your agent execution frequency and processing time while ensuring SLAs are processed efficiently on dedicated resources. I recommend implementing these changes in a development environment first and monitoring the results before moving to production.
References:
- Pega-ProCom agents
- Avoiding race conditions for advanced agents
- Queue processing for SLAs
- Node classification