Hi,

About the Node Types list, one of the articles from Steve Mekkelsen Madden gives the following answer :

Node Types provided by Pega 7.3

Starting with Pega 7.3, there are a total of 11 different Node Types that you can use to define the nodes in the Pega cluster of your deployment.

There is no effect on the Dynamic System Settings or prconfig.xml settings when you use a Node Type.

1. Universal
2. WebUser
3. BackgroundProcessing
4. BIX
5. Search
6. RunOnAllNodes
   You cannot start a node with this Node Type. Agents and Listeners can be mapped to this Node Type so that they run on all nodes in a cluster.
7. Custom1
8. Custom2
9. Custom3
10. Custom4
11. Custom5

Hi all,

following last session about agents i'd like to share a scenario about managing high workload.
Say for example we have a queue for a standard agent where each item processing requires about 30 seconds.
Those queue items are unrelated to each other, meaning we don't have to process them in a FIFO logic 
This queue gets populated with thousands of records each day. So the challenge is to be able to keep up with all the workload.
Our production environment consist of 3 nodes. So the obvious solution is to have the agent run on each node and that means we can manage 8640 queue item in a day.
What if our load is 20K queue item per day?

Session proposed solution:
- Have multiple agent running on the same queue: say your queue is System-Queue-MyQueue. When invoking queue-for-agent API you can specify different agents in a round robin fashion (Agent1, Agent2, Agent3)
Each one of those agent is fetching items from System-Queue-MyQueue so you would have 3 times the processing power (8640*3)
The con here is that we can can't scale on the load without changing the queue logic. Meaning that if we have to add a new agent working on the same queue (say Agent4) we need to update our round robin queue logic and create the new agent.
This requires a release cycle and can't be done by the production team managing the environment

- Leveraging data flow component which has multi threading capabilities: this solution has been proposed in the session but i'm not familiar with it and can't elaborate much on it for now. Maybe the person that shared it can pop in and detail it a bit

So, what's your take on this one?

Few questions on this use case:-

who write the data into queue items? is it some external API is calling Pega API to capture the data and process it later?

can you capture the data into an external table? - Fully exposed table i.e. table with all exposed columns and without blob/clob columns

Looking at the use case high level, Dataflow Batch run can easily support this.

Dataflow batch Requirements:

1. All the data should be in a fully exposed table – BLOB/CLOB columns are not supported
2. To support parallel processing we need to have a column in the input table which acts as a Partition Key for e.g. a column name “Key” holds values between 1- 10 where all the records in the input table are associated
3. The parallel process of Dataflow Batch is determined with the number of unique partitions input table have. As mentioned in step 2, 10 thread can process the Batch in Pega cluster.

Dataflow Design:

1) New Pega Data class: Data-XXX point to the external table - this is where the current queue items data captured will be stored

2) Make Pega API or any other process capture the data and writes into Data-XXX

3) Create a new Dataset of type Database Table under --> Data-XXX

4) Create a Dataflow under Data-XXX as a source with the Dataset created at step 3

5) in Dataflow add the filters to process the data based on different conditions

6) Dataflow Destination can be configured to call an Activity to perform the action business logic needs

Configuration:

1) Single Advanced Pega agent which run the Dataflow Batch run periodically, you need to ensure that Batch run is started with only one node

2) Decision --> Services Landing page allows configuring the number of threads that batch can parallel process the load.