Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

i have a web directory that has many folders and many sub folders containing files.

i need to download everything using wget or bash.

share|improve this question

migrated from Feb 4 '10 at 1:42

This question came from our site for professional and enthusiast programmers.

Try: wget -r and see if that works.

share|improve this answer

The best way is:

wget -m <url>

Which is short for wget "mirror":

  -m,  --mirror             shortcut for -N -r -l inf --no-remove-listing.
share|improve this answer

wget --recursive (or whatever) didn't work for me (i'm on CentOS). lftp did it:

 lftp -c "open http://your.server/path/to/directory/; mirror"
share|improve this answer
The "whatever" flags are quite important... yes, wget's flags are a bit over the top, but what do you expect from a swiss army knife. – vonbrand Feb 24 '13 at 19:50
$ wget \
 --recursive \
 --no-clobber \
 --page-requisites \
 --html-extension \
 --convert-links \
 --restrict-file-names=windows \
 --domains \
 --no-parent \

This command downloads the Web site

The options are:

  • --recursive: download the entire Web site.
  • --domains don't follow links outside
  • --no-parent: don't follow links outside the directory tutorials/html/.
  • --page-requisites: get all the elements that compose the page (images, CSS and so on).
  • --html-extension: save files with the .html extension.
  • --convert-links: convert links so that they work locally, off-line.
  • --restrict-file-names=windows: modify filenames so that they will work in Windows as well.
  • --no-clobber: don't overwrite any existing files (used in case the download is interrupted and resumed).

Link to source

Or try solution from ask Ubuntu.

share|improve this answer

See Wget Recursive Retrieval.

wget -r -l 5 -O whatever
share|improve this answer

You have a web directory? Is it situated on a remote machine and you can only access it through HTTP, or do you have shell access? Your mention of bash implies shell access, unless you mean using wget from the bash prompt.

Wget is not always very efficient so if you have shell access to the machine where the web directory is located and you want to download it, you could do this

$ tar cjf webdir.tar.bz2 webdir 

and then transfer the archive with ftp or scp.

share|improve this answer

You could also try the following if you have an FTP account:

lftp USER:PASSWORD@FTPSERVER -e "mirror&&exit"
share|improve this answer

You must log in to answer this question.