I want to download a data file from a remote ftp server to my machine at regular intervals. Once the file is downloaded I want to call another script which will process the file.
My development machine is mac os x, the eventual deployment environment is linux.
there is a remote public ftp server run by the weather bureau, on which the weather bureau post files containing weather forecasts. This is not my machine, I can only download from it. The weather bureau puts a new file here every six hours, the time is part of the file name. I will have a tool on my machine that can process the data in these files. I am not asking for help developing this tool. I am asking for help gluing the pieces together, scheduling the downloads and executions of the tool.
what I want to do:
every six hours i want to download the new file from the weather bureau ftp server. if this fails I want to: send an email and reschedule the download.
if the download is still failing after six hours I want to stop trying to get this file, as there should now be a new file with a new name.
Once the file is successfully downloaded, I want to call my tool to process the file. If the tool fails because the file is corrupt it will return an error, and I want to reschedule the file for download
What's would be the stock standard way to automate this?
I know I can use cron to schedule curl to download and to run a script that will process the downloaded file at regular intervals, and I know could write a slightly more complex script or an application that would do this and add error handling, rescheduling and sending status emails.
But one of my requirements for this project is to write as little custom code as possible, instead I should try to use standard, tried and true existing tools, and if I do have to write code, to try and write the most straightforward code possible. The reason for this is the code will potentially be installed on a large number of machines, all of which will need to be tweaked, customised and maintained by different people, long after I am gone from the project, so the intention is to use well documented, well supported tools as much as possible.
This seems such a common task, there must be tools and scripts all over the internet, written by people who have carefully considered everything that could possibly go wrong when you need to download and process a file from a remote server at regular intervals, with error handling, rescheduling and sending status messages.
Is that what 'Expect' is for?
What would you recommend?