Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I use the following code to download "example1.com" with Windows. However, this code is limited to only one URL.

Is it possible to import a txt file which contains several URLs instead?

wget -erobots=off --no-parent --wait=3 --limit-rate=20K -r -p -U "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" -A htm,html,css,js,json,gif,jpeg,jpg,bmp http://example1.com
share|improve this question
    
You could write yourself a batch script which reads the txt file and invoced wget for each line. Don't really know about batch script because I'm more of a Linux guy ;-) –  André Stannek Mar 28 '13 at 20:28

2 Answers 2

I found the easiest way: wget has a command to import links from a text file.

wget -i mylist.txt
share|improve this answer

You can do that with bash, the Windows installer can be found here (when asked for components to install choose only MSYS), on Linux you probably have it as default command line interpreter so you just run console emulator and you're ready to go. To run it on Windows open C:\MinGW\msys\1.0\msys.bat.

Let's say your URL database is called db.txt and it has each address in a separate line, with no spaces. Then the command to run wget for each address will be:

for line in `cat db.txt`; do wget -erobots=off --no-parent --wait=3 --limit-rate=20K -r -p -U "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" -A htm,html,css,js,json,gif,jpeg,jpg,bmp $line; done

Notice that I've used accent characters (those to the left of the 1 button), not aphostrophes. Before running this command you have to change your directory to the folder where the file is:

cd /home/YourUsername
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.