Robotics Automation Studio 8: reading in parallel from a share drive
Our customer plans on running instances of the Robotics runtime on multiple client machines, with each machine polling a share drive to see if new work has come in. The files arrive in CSV format, which the automation will copy to it's own local drive, open and read the file, and input the contents into a locally installed application. The trick comes in co-ordinating the share drive access.
We configured the automation to monitor the share drive and want the first one to detect OnCreate to move the file to its own local. However, multiple clients detect OnCreate simultaneously, with all but one failing the Move because the file is no longer there (or in use). What is the standard best practice for handling this situation?
Of course we can create a round-robin shell script to dole the CSV files out to the clients, but because the files take different times to process we would rather a mechanism whereby machines which are idle go to the share and pick up work.
**Moderation Team has archived post**
This post has been archived for educational purposes. Contents and links will no longer be updated. If you have the same/similar question, please write a new post.
Accepted Solution
Pegasystems Inc.
JP
Necessity is the mother of... automation solutions.
I managed to implement this by utilizing the Windows file locking mechanism as follows.
Any process which wants to pick up the input file has to first get hold of a lock file; only one will succeed and the try-catch will exception out all the rest. The random timeout at the beginning should no longer be needed but I left it in there to reduce the likelihood of exceptions (don't think it makes any difference). The CloseStream script does nothing but "stream.Close()" - Jeff let me know if there's a component that does this.
Pegasystems Inc.
US
You could build a robot to create a new case with the file attached when it arrives and route that case to a Workbasket. You could then work each file using robots attached to the Robotic Console.
Without some kind of routing mechanism you will always have contention for new files as they arrive.
Pegasystems Inc.
JP
Work basket - what's that in the Openspan world? How would the various machines get it from that work basket without the same contention as the shared drive?
Pegasystems Inc.
US
Take a look at this video on TechTalk which discusses using the Robotic Console and Robotic Automation.
Pegasystems
US
>>> How would the various machines get it from that work basket without the same contention as the shared drive?
If you're talking about the standard Pega concept of moving an assignment from a work basket to either a user list or another work basket, the contention is normally handled with standard work object locking.
That is, each contender tries to lock the work and only one wins.
To see the details, try one of the standard parts of a Pega flow rule that routes the assignment, and use the Pega tracer to see what activities it calls, and you'll see the locking occurring within one of those activities.
/Eric
Pegasystems Inc.
JP
Hi Jeff/Eric, thank you for your replies. They both seem very Pega centric; are they based on the assumption that the customer has installed Robotics in conjunction with PRPC 7.2.1+? Because our customer has only installed Robotics 8 standalone - and I didn't find where the console is in that....
Pegasystems Inc.
US
Michael,
I was trying to give you some ideas that were best practice. To solve your issue of contention you would need a router to dispense the work. That could be a round robin script, a single robotic process that listens for new files or a number of other methods. Having each process try to grab the new file is problematic.
Jeff
Pegasystems Inc.
US
Pega Robotics alone doesn't provide a central server or anything to queue work. Best practice would be to use the Pega Robotics Console. One way we have used before is basically what Jeff has suggested; creating a separate "bot" to assign work to each "worker bot".
Pegasystems Inc.
JP
I'm still not sure what "Pega Robotics Console" refers to. I was initially wondering if there was a Robotics equivalent to the PRPC work baskets - not intending to install PRPC just to use its work baskets! If not, I can write a robot to distribute work to different physical locations.
Pegasystems Inc.
US
Here is an article that will help you understand more about the Pega Robotics Automation Management Console.
Pegasystems Inc.
JP
This one's for you Jeff:
I tried to implement some sort of mechanism which uses a contention mechanism built into Windows, to save from doing it myself. I started with this one (you will recognize the random script):
The random pause is to stagger move attempts to reduce the likelihood of different machines attempting the move at the same time, and the try-catch (ignoring the exception) is to ignore "file not exist" or "file in use" exceptions. By large, this works quite well but there is a small incidence of two machines successfully moving the same file into their own local drives. So Microsoft didn't built contention handling into their move command perfectly.
Is it possible to use the file lock mechanism instead? It is not possible for two machines to get a write lock on the same file at the same time - perhaps each machine could first try to get the file lock (with the unsuccessful ones giving up on exception) then moving the file after that?
This one's for you Jeff:
I tried to implement some sort of mechanism which uses a contention mechanism built into Windows, to save from doing it myself. I started with this one (you will recognize the random script):
The random pause is to stagger move attempts to reduce the likelihood of different machines attempting the move at the same time, and the try-catch (ignoring the exception) is to ignore "file not exist" or "file in use" exceptions. By large, this works quite well but there is a small incidence of two machines successfully moving the same file into their own local drives. So Microsoft didn't built contention handling into their move command perfectly.
Is it possible to use the file lock mechanism instead? It is not possible for two machines to get a write lock on the same file at the same time - perhaps each machine could first try to get the file lock (with the unsuccessful ones giving up on exception) then moving the file after that?
- How do we get a lock in Studio?
- How do we then release that lock to move the file without the gap in between?
Incidentally I took a quick look at your article on Pega Robotics Automation console (sorry didn't have time to look further) - am I correct in saying it requires PRPC?
Pegasystems Inc.
US
The Pega Robotics Automation console does require PRPC. You can get your work to PRPC through automation, however. The Case component is designed to add cases to PRPC. These cases can then be handed to robot workers through the robotics console.
Accepted Solution
Pegasystems Inc.
JP
Necessity is the mother of... automation solutions.
I managed to implement this by utilizing the Windows file locking mechanism as follows.
Any process which wants to pick up the input file has to first get hold of a lock file; only one will succeed and the try-catch will exception out all the rest. The random timeout at the beginning should no longer be needed but I left it in there to reduce the likelihood of exceptions (don't think it makes any difference). The CloseStream script does nothing but "stream.Close()" - Jeff let me know if there's a component that does this.
Pegasystems Inc.
US
Please attach your image as a separate file. The inline images get compressed and hard to see. There is no component that Pega Robotics provides to do this.