Download a list of files and use domain name as filename

Question

I got a bunch of files to download which works great with

wget -i list_of_urls

but my problem is, that wget uses the filename of the downloaded file. Is there a way (or a different tool) which is able to use the whole url als filename, f.e.

http://www.example.com/file1.html
http://www.example.com/file2.html

Lead to the files:

http___www_example_com_file_1.html
http___www_example_com_file_2.html  

Answer 1

Use some simple bash scripting. For example, if you have a file "foo" with the URLs:

http://www.google.com/index.html
http://www.cnn.com/index.html

You can run:

for i in `cat foo`; do wget $i -O `echo $i | sed 's/[^A-Za-z0-9]/_/g' | sed 's/_html$/.html/'`; done

which produces

http___www_cnn_com_index.html
http___www_google_com_index.html

Answer 2

Use the "-x" option documented here. For example, given a file "foo" with contents:

http://www.google.com/index.html
http://www.cnn.com/index.html

If you run

wget -x -i foo

then you will get these files:

www.google.com/index.html
www.cnn.com/index.html