Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to download a data file from a remote ftp server to my machine at regular intervals. Once the file is downloaded I want to call another script which will process the file.

My development machine is mac os x, the eventual deployment environment is linux.

in detail:

there is a remote public ftp server run by the weather bureau, on which the weather bureau post files containing weather forecasts. This is not my machine, I can only download from it. The weather bureau puts a new file here every six hours, the time is part of the file name. I will have a tool on my machine that can process the data in these files. I am not asking for help developing this tool. I am asking for help gluing the pieces together, scheduling the downloads and executions of the tool.

what I want to do:

every six hours i want to download the new file from the weather bureau ftp server. if this fails I want to: send an email and reschedule the download.

if the download is still failing after six hours I want to stop trying to get this file, as there should now be a new file with a new name.

Once the file is successfully downloaded, I want to call my tool to process the file. If the tool fails because the file is corrupt it will return an error, and I want to reschedule the file for download

What's would be the stock standard way to automate this?

I know I can use cron to schedule curl to download and to run a script that will process the downloaded file at regular intervals, and I know could write a slightly more complex script or an application that would do this and add error handling, rescheduling and sending status emails.

But one of my requirements for this project is to write as little custom code as possible, instead I should try to use standard, tried and true existing tools, and if I do have to write code, to try and write the most straightforward code possible. The reason for this is the code will potentially be installed on a large number of machines, all of which will need to be tweaked, customised and maintained by different people, long after I am gone from the project, so the intention is to use well documented, well supported tools as much as possible.

This seems such a common task, there must be tools and scripts all over the internet, written by people who have carefully considered everything that could possibly go wrong when you need to download and process a file from a remote server at regular intervals, with error handling, rescheduling and sending status messages.

Is that what 'Expect' is for?

What would you recommend?

share|improve this question
Perhaps this will get a better response from stackoverflow 'cos they know about tools related to programming. And if migrating it there, i'd suggest ideally if possible, somehow doing it while compound(the question's author) is awake and online 'cos things get replied to fast over there, and then many people could be onto something else! – barlop Feb 14 '11 at 23:47
@barlop: I'm not sure this is a pure programming requirement - more likely a scripting one to string together a standard set of commands. – Linker3000 Feb 15 '11 at 0:55
@Linker3000 they cover that too though – barlop Feb 15 '11 at 4:18

Hmm, that's a broad, general spec so it;s hard to be precise.

  • Will all the machines providing the data be running Linux? If so, maybe look at using rsync (or scp) to pull down the data files as this will reduce the need for so much scripting of the file transfer (not that there would be much) and possibly eliminate the need for 'expect', which can often be tripped up by an unplanned message or event.

  • You don't say what 'processing' will be needed on the downloaded/transferred files so that's hard to comment upon.

  • Rescheduling could be handled by cron or, in the event of a transfer error, just reading rsync or scp's exit status may be enough.

  • How are status messages to be sent? If via email then you could use the standard, command-line 'mail' command or there is a lovely tool that can send SMS messages via an attached modem.

On the face of it, it looks as if a small amount of bash-type scripting would hang together enough standard commands to do what you want so that you won't need to borrow code from elsewhere, but if you can elaborate you'll probably get more replies.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .