I am using API that has some limit of requests in an hour. But my script does all at one time so I lose about 1/3 of requests because I get 403.

Is there any way to check response of wget and if I get 403 to wait 5 mins and retry?

And here is my (for now) test code:

system ("wget \"http://test-link.com/403/\" -O  {$dir}/{$in_dir_counter}.xml");
$test = system ("wget \"http://test-link.com/403/\" -O  {$dir}/{$in_dir_counter}.xml");

echo "responsed - ".$test;      

both returns same.

  • What does your own research suggest? – Dave Nov 7 '13 at 10:48
  • My research? All forums I've read suggests to add timeout for EVERY request. But I can't do it because with such conditions (403) it takes 1-2 days to complete. So if I add like 10sec timeout it would be atleast 4-5 days in best wishes. – user270181 Nov 7 '13 at 10:52
  • It would be helpful if you posted your script or the relevant part of it – Tog Nov 7 '13 at 10:58
  • Just added part of code. Hope it helps. – user270181 Nov 7 '13 at 11:04

How about using a simple script for that:

  • Run the script once every 5 minutes unless it's running already.
  • Check the age of the local file. If it's older than a specific threshold, download it again.

So if everything wents smooth, nothing happens, unless a file is outdated. If a file is outdated and failed downloading, you can retry next Iteration.

I'm not sure why you tagged this with php, but if you're actually running a php script this approach is rather easy to do (given you've got web sockets enabled):

foreach($files as $file)
    if (@filemdate($local_path + $file) + $cache_duration < time())
        @copy($remote_path + $file, local_path + $file);

Note that $remote_path can indeed be a HTTP or FTP URL. There's no need to invoke wget. The @ will prevent error messages to be printed.

To prove that this won't cause unneeded waiting:

  • Assume you've got 1000 files to download, but you can only download up to 250 files per hour.
  • Set cache_duration to a save time where you'll get all files, like 24h (24 * 60 * 60).
  • Rerun the script above once every hour.
  • The first iteration the first 250 files will be updated. The others will fail.
  • The second iteration the first 250 files will be skipped (due to being recent enough) and the next 250 files will be downloaded.
  • After the fourth iteration you'll have all 1000 files updated/downloaded.
  • Of course you can set a shorter intervall, like 5 minutes, but this will create a lot more requests/traffic (depends on whether this is acceptable).

Alternative script idea:

  • Try to download a file.
  • If it fails, you should be able to determine that based on wget's return value/exit code. So in that case wait 5 minutes, then repeat.
|improve this answer|||||
  • Like i said in comments - it already takes 1-2 days. If I add 5min timeout or start another one it won't help but make it worse. I need to get all request with minimal time waste. – user270181 Nov 7 '13 at 11:06
  • My first approach won't use any timeout. It will try to download all outdated files at once. 5 minutes later you retry, only downloading the files that failed the first attempt. There is only some waiting between attempts downloading everything. – Mario Nov 7 '13 at 11:08
  • as i said all script works 1-2days. It depends on how much info they give me. Sometimes its 15k, sometimes 45k. And I have 2 more scripts. All work in chronological order. So I can't waste time to check all files and retry it. I have to do it in the process. – user270181 Nov 7 '13 at 11:13
  • How about parallelizing it then? You could as well store some pointer or index to the latest file you could retrieve last attempt. Just get a bit creative here. This way you don't have to recheck all files every time; just reset the index once everything is done or so. – Mario Nov 7 '13 at 11:15
  • Right now i am thinking to check a file (where i write info). Because every request creates new file. If the file size is 0 (403) then sleep 5 sec and try again. What do you think about that? – user270181 Nov 7 '13 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.