Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I use the following code to download "" with Windows. However, this code is limited to only one URL.

Is it possible to import a txt file which contains several URLs instead?

wget -erobots=off --no-parent --wait=3 --limit-rate=20K -r -p -U "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" -A htm,html,css,js,json,gif,jpeg,jpg,bmp
share|improve this question
You could write yourself a batch script which reads the txt file and invoced wget for each line. Don't really know about batch script because I'm more of a Linux guy ;-) – André Stannek Mar 28 '13 at 20:28

I found the easiest way: wget has a command to import links from a text file.

wget -i mylist.txt
share|improve this answer

You can do that with bash, the Windows installer can be found here (when asked for components to install choose only MSYS), on Linux you probably have it as default command line interpreter so you just run console emulator and you're ready to go. To run it on Windows open C:\MinGW\msys\1.0\msys.bat.

Let's say your URL database is called db.txt and it has each address in a separate line, with no spaces. Then the command to run wget for each address will be:

for line in `cat db.txt`; do wget -erobots=off --no-parent --wait=3 --limit-rate=20K -r -p -U "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" -A htm,html,css,js,json,gif,jpeg,jpg,bmp $line; done

Notice that I've used accent characters (those to the left of the 1 button), not aphostrophes. Before running this command you have to change your directory to the folder where the file is:

cd /home/YourUsername
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .