How to recursively download an entire web directory?

Question

i have a web directory that has many folders and many sub folders containing files.

i need to download everything using wget or bash.

Answer 1

Try: wget -r and see if that works.

Answer 2

The best way is:

wget -m <url>

Which is short for wget "mirror":

  -m,  --mirror             shortcut for -N -r -l inf --no-remove-listing.

Answer 3

See Wget Recursive Retrieval.

wget -r -l 5 -O whatever http://example.com/

Answer 4

"curl is a command line tool for transferring data with URL syntax, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS"

link text

Answer 5

you could use wget to mirror the directory structure. Did that with my prof's website on a daily basis. very handy ;-)

Answer 6

You have a web directory? Is it situated on a remote machine and you can only access it through HTTP, or do you have shell access? Your mention of bash implies shell access, unless you mean using wget from the bash prompt.

Wget is not always very efficient so if you have shell access to the machine where the web directory is located and you want to download it, you could do this

$ tar cjf webdir.tar.bz2 webdir 

and then transfer the archive with ftp or scp.

Answer 7

You could also try the following if you have an FTP account:

lftp USER:PASSWORD@FTPSERVER -e "mirror&&exit"