I need to download 20 files each of almost 2 GB each.
However, more often than not my downloads are interrupted either by internet failure or VPN errors. So, If its a download error, I need to restart the download or if its a VPN error, I need to relogin in openVPN and restart the download.
I want to automate this process, how do I do it?
I am aware there is iMacros for firefox but its only useful for activities like form filling or can it be extended to such tasks as well?
I have made a simple procedure/ algorithm for this. Can someone help me figure out how I could implement it?
1. Connect to OpenVPN
2. Go to Link from document*
3. Save Link
4. If download interrupted, Do 2
5. If VPN interrupted, Do 1
6. If download complete, Link= Link + 1
7. If all Links finished, End task.
Document: File which contains links to be downloaded.
Add a comment
|
1 Answer
If you have ssh access to the server, I suggest using rsync. It can determine which files need to be downloaded, in whole or in part. If the VPN fails you can restart it and then rerun the rsync command; it will pick up where interrupted.
If you need to download over http then I suggest using wget. It can similarly continue downloading partially downloaded files with retries but I'm not sure how good it is about determining whether a file it already has is up to date.
