My idea is to keep an active list of files in that directory. then check the directory periodically and if there is a change, download / move the file.

Hi,

Apply MonitorAll properly on web adapter, it will monitor all browsed based applications and when sharepoint website found then it will hook the application. You can automate the flow like when user clicks add button or upload button then you can detect the file which is uploaded to sharepoint.

Share point has an alert function. It can send emails when an item is added to a library. The email sent contains a link to the new file. So you could have an email setup and use outlook to watch for new files without having to deal with the actual share point site itself.