Authored by <Anonymous>

Background

Starting with Pega Platform version 8.1, Job Scheduler and Queue Processor rules replace Agents and improve background processing. More to the point, Job Scheduler rules replace Advanced Agents for recurring or scheduled tasks, and Queue Processor rules replace Standard Agents for queue management and asynchronous processing.

From the logical perspective, using Queue Processor rules involve three roles:

1. The Producer, which publishes messages to a Queue Processor rule. For example, we can publish messages to a Queue Processor rule using the Queue-For-Processing method in an Activity.
2. The Stream Service, which persists and replicates the messages we publish to Queue Processor rule. As illustrated in the diagram below, the Stream service is built on the Apache Kafka platform.
3. The Consumer, implemented as an activity, which subscribes to messages published to a Queue Processor rule.

Configure Stream Service

Stream service is an essential component for healthy and resilient background processing. It is important to configure and monitor the service properly, as described in the Deploying and Operating Stream Service guide.

Configure Node Types

Node classification is the process of separating nodes in a cluster, segregating them by purpose, and predefining their behavior. Queue Processors and Job Schedulers run on BackgroundProcessing node types by default to avoid performance issues for users. It is essential to plan node types topology properly:
• Dedicate at least two, but not more than four nodes as Stream. • Do not mix Stream nodes with other nodes types. • Dedicate one or more nodes for BackgroundProcessing node type. • Do not use Universal or untyped nodes.

Monitor system health

Monitor Pega instances by associating a health check with every node in a cluster. Refer to the Pega Platform documentation for more details. Unhealthy nodes must be taken out of service and restarted.

Best-practices for writing background processing activities

Queue processors, job schedulers, and agents are background jobs that use an activity to implement processing logic. These activities are expected to finish processing in less than a few seconds. If an activity takes longer to complete, use the following best practices to ensure that work items are not affected during the shutdown process.

Create small work items

Break down large work items into smaller work items, queue them to a queue processor, and process them individually. Use a queue processor instead of an agent because queue processors provide better performance, resiliency, fault tolerance, and flexibility than agents. Scale queue processors horizontally by increasing the number of nodes configured with the node type. Scale queue processors vertically by increasing the number of threads.

Use checkpoints

To ensure that activity logic can gracefully handle restarts without corrupting data or introducing inconsistencies into your application, use checkpointing by saving the state of jobs in persistent storage or as messages in a queue that is appropriate to the use case.

For example, consider a scenario where a work object can have the following statuses: NEW, IN-PROGRESS, and PROCESSED. You can read all of the work object records to be processed, loop through them one by one, mark the status of the current work object as IN-PROGRESS, continue with the business logic, and when processing is finished, change the status to PROCESSED.

Divide and scale tasks

If you have background tasks that can be broken down into multiple tasks that have different performance capabilities, consider dividing them and scaling each independently. Queue processors can be scaled horizontally by increasing the number of relevant nodes and can be scaled vertically by increasing the number of threads. For more information, see Managing queue processors in Admin Studio help.