How about using a simple script for that:
- Run the script once every 5 minutes unless it's running already.
- Check the age of the local file. If it's older than a specific threshold, download it again.
So if everything wents smooth, nothing happens, unless a file is outdated.
If a file is outdated and failed downloading, you can retry next Iteration.
I'm not sure why you tagged this with php
, but if you're actually running a php script this approach is rather easy to do (given you've got web sockets enabled):
foreach($files as $file)
if (@filemdate($local_path + $file) + $cache_duration < time())
@copy($remote_path + $file, local_path + $file);
Note that $remote_path
can indeed be a HTTP or FTP URL. There's no need to invoke wget. The @
will prevent error messages to be printed.
To prove that this won't cause unneeded waiting:
- Assume you've got 1000 files to download, but you can only download up to 250 files per hour.
- Set
cache_duration
to a save time where you'll get all files, like 24h (24 * 60 * 60
).
- Rerun the script above once every hour.
- The first iteration the first 250 files will be updated. The others will fail.
- The second iteration the first 250 files will be skipped (due to being recent enough) and the next 250 files will be downloaded.
- After the fourth iteration you'll have all 1000 files updated/downloaded.
- Of course you can set a shorter intervall, like 5 minutes, but this will create a lot more requests/traffic (depends on whether this is acceptable).
Alternative script idea:
- Try to download a file.
- If it fails, you should be able to determine that based on wget's return value/exit code. So in that case wait 5 minutes, then repeat.