Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

There's this beautiful boy who periodically uploads pictures of himself to his website. I am trying to automate the process of downloading these images to my computer.

So far, I'm able to download his webpage and parse it for jpg files. I end up with a file like this.

http://stat.ameba.jp/user_images/20120129/19/maofish/f9/60/j/o0480064011762693689.jpg
http://imgstat.ameba.jp/view/d/70/stat001.ameba.jp/user_images/20120127/22/maofish/f7/3e/j/t02200293_0480064011759076335.jpg
http://imgstat.ameba.jp/view/d/70/stat001.ameba.jp/user_images/20120125/18/maofish/80/46/j/t02200293_0480064011755033425.jpg
http://imgstat.ameba.jp/view/d/70/stat001.ameba.jp/user_images/20120120/20/maofish/3c/99/j/t02200290_0480063311745603530.jpg
http://stat.ameba.jp/user_images/20100219/16/maofish/33/0b/j/t01400198_0140019810420649113.jpg
http://stat.ameba.jp/user_images/b0/09/10101851128_s.jpg
http://stat.ameba.jp/user_images/9c/26/10027225053_s.jpg

I can get any of those images by doing a wget on them, but I would like to automate this process to get everything in the list. I tried piping and redirecting to wget, but it doesn't work. How can I accomplish what I'm trying to do?

share|improve this question
    
What have you already tried? – r4. Jan 30 '12 at 9:10
1  
I have tried wget | list and wget < list. – tony_sid Jan 30 '12 at 9:12

You can use -i option of wget such as:

$ wget -i input_file.txt

You will get all files downloaded in the current directory. You can see man wget for more options.

share|improve this answer
    
Is there a way to tell wget to ignore images that are smaller than a certain size? – tony_sid Jan 30 '12 at 9:22
1  
If the target is to remove the small files (less than some threshold), you can use find to delete them automatically after download. – Khaled Jan 30 '12 at 9:26

minsize="50" #grab all over 50kb

for x in $(cat list)
do
if [ "$(echo $(GET -Ssed $x | grep Length | awk '{print $2}') /128 |bc)" -ge $minsize ]; then
wget -q $x
fi
done

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .