Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

UPDATED:

I've found using this Bash script fixes the problem of having GIF files with a .jpg extension.


I'm attempting to download images from a forum who's url uses the following format:

http://www.someforum.com/attachment.php&id=XXX

I wrote a bash script that uses wget to retrieve these images:

for i in {1..10}

do
    wget --accept .jpg,.jpeg --cookies=on --load-cookies=cookies.txt -p "http://www.someforum.com/attachment.php&id=${i}" -O "image${i}.jpg"

done

It works and downloads the images. However if there isn't an image it still downloads the resulting HTML and stuffs it in XX.jpg.

Curl does the same:

for i in {1..10}

do
    curl --cookie cookies.txt "http://www.someforum.com/attachment.php&id=${i}" -o "image${i}.jpg"

done

Is there anyway to reject results that are not /image/*? Right now I am assuming that the images are jpeg, it would be nice to detect the MIME/TYPE and use the appropriate filename.

Finally, wget is giving 500 response codes when an image isn't found, if I can filter 200 response codes this may yield a solution.

Bash, Ruby, Python answers are acceptable.

share|improve this question
up vote 4 down vote accepted

wget returns a non-zero exit code on error; it specifically sets exit status == 8 if the remote issued a 4xx or 5xx status. So, you can modify your bash loop to unlink the file if wget doesn't exit with success:

for i in {1..10}
do
    wget --accept .jpg,.jpeg --cookies=on --load-cookies=cookies.txt -p "http://www.someforum.com/attachment.php&id=${i}" -O "image${i}.jpg" || rm "image${i}.jpg"
done

Similarly, curl has a --fail option, with which it wont save the file and returns exit status 22 when the http status is >= 400.

share|improve this answer
    
This works, thank you. I'm going to leave the question open for a while to see if anybody comments regarding MIME/Content-Type as I'm still (understandably) getting GIF files saved as .jpg – Ash Mar 18 '12 at 11:18
1  
Ah, I didn't realize you had files saved with wrong extension problem too. I would have suggested file --mime to map the appropriate real extension too. – dbenhur Mar 18 '12 at 17:43
    
I was hoping to avoid post processing but having looked through the entire wget manual it seems as though it's the only way. – Ash Mar 18 '12 at 22:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .