Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I save images with:

#!/bin/bash
for i in {1..30000}
do
    wget "http://services.runescape.com/m=itemdb_rs/3809_obj_sprite.gif?id="$i
done

Is there a way to speed up the process? Sometimes a URL does not exist and I get:

connected.
HTTP request sent, awaiting response... 404 Not found
2012-08-04 18:09:36 ERROR 404: Not found.

How can I continue after this error?

share|improve this question
    
If you have an additional question about how to download out only certain object, please ask an additional question. Mixing two subjects in one question just over-complicates the matter :) –  Oliver Salzburg Aug 4 '12 at 16:26
    
@OliverSalzburg - Your comment is spot on. But you chose to edit the title to reflect the question you answered. Neglecting to notice that there where two answers answering his other question. –  Nifle Aug 4 '12 at 16:37
    
@Nifle: Feel free to read the history of the question and change it if you feel that I've missed the point with my edit. Also, please note that I edited and commented on the question before I wrote my answer. –  Oliver Salzburg Aug 4 '12 at 16:39
add comment

3 Answers

Try

wget "http://s.../m=itemdb_rs/3809_obj_sprite.gif?id="$i || true

This way, if wget fails, the result of that line is still zero and your script continues.

To learn how to properly handle errors in a bash script, check out the following resources:

share|improve this answer
    
+1 This is a good suggestion! –  Canha Aug 4 '12 at 17:07
add comment

You can't. There is no way to know in advance what will files are going to be missing

You can however try to fail more quickly by adding these flags --tries=1 --connect-timeout=5

share|improve this answer
    
Can I "multi thread" it? PHP script with same while loop performs MUCH faster. –  Ultra Aug 4 '12 at 16:26
    
@Ultra - Well then use PHP instead of wget. Or do the obvious thing and run multiple instances of the script with different ranges. –  Nifle Aug 4 '12 at 16:29
    
Of course you can "multithread" in a bash loop. @Ultra –  slhck Aug 4 '12 at 16:30
add comment

You could try something like this.

#!/bin/bash
for i in {1..30000};
do 
x="http://64.79.147.130/m=itemdb_rs/3809_obj_sprite.gif?id="$i
if [[ $(echo $(GET -sd $x | grep "404")) = "" ]]; 
then 
wget --no-dns-cache -nc -U"Firefox/10" $x 2>&1 \
| grep "Saving" | sed 's,Saving to,Saved,'; 
fi 
done
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.