Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Not really a question this one - but alternative answers are certainly welcome! :)

Sometimes I need to download files listed in an Apache directory listing, say as in:

... in a given directory on my computer, and I'd like to use command-line tools for that.

As far as wget is concerned, actually there are some switches that should be in place - noting that wget tends to either download single items - or to reconstruct the server folders locally!

As I just spent some half an hour figuring out what those proper switches are (to copy the remote files locally as through, say, FTP), I'd like to document them here; so the above link would be downloaded with:

wget -nd -r -l 1 http://www.gnu-darwin.org/www001/src/ports/net/samba-libsmbclient/work/samba-3.0.28/source/libsmb/

... where:

   -nd
   --no-directories
       Do not create a hierarchy of directories when retrieving
       recursively.  With this option turned on, all files will get saved
       to the current directory, without clobbering [...]

   -r
   --recursive
       Turn on recursive retrieving.

   -l depth
   --level=depth
       Specify recursion maximum depth level depth.  The default maximum
       depth is 5.

It would be nice to hear alternatives to the above command - maybe using different set of switches for wget - or maybe using curl or other packages...

share|improve this question
    
Also related: unix.stackexchange.com/questions/36798/… –  sdaau Oct 8 at 9:18

2 Answers 2

See also -nH a.k.a --no-host-directories and --cut-dirs options. I also frequently use --accept/-A and --reject/-R.

share|improve this answer
    
Many thanks for these options, @jw013 - great to have a quick reference for them! Cheers! –  sdaau Sep 19 '11 at 19:24

After some time, thanks to the pointers by @jw013 and in Sync with a Directory Listing of An Apache Server - Unix & Linux Stack Exchange, I finally found my "ultimate" wget command.

Assume you have a tree of files and directories in a directory called myfiles_dir, and let that directory be accessible through an Apache directory listing on http://localhost/shared/myfiles_dir. Then, you can sync directory locally with:

wget -r -N --no-parent --reject '*index.html*' -nH --cut-dirs=1 http://localhost/shared/myfiles_dir/

... where:

-r,  --recursive              specify recursive download.
-N,  --timestamping           don't re-retrieve files unless newer than
                              local.
-np, --no-parent              don't ascend to the parent directory.
-R,  --reject=LIST            comma-separated list of rejected extensions.
-nH, --no-host-directories    don't create host directories.
     --cut-dirs=NUMBER        ignore NUMBER remote directory components.

This will download the contents into a subdirectory myfiles_dir created in the directory wget was called from (the working directory), without any residual index.html files.

Note that the trailing slash / after the http link/address is extremely important: if it is not there - as in http://localhost/shared/myfiles_dir - upon repeated calls of wget from the same local working directory, the HTML of the directory listing will be saved as multiple copies e.g. myfiles_dir.1, myfiles_dir.2 etc, in spite of any switches (although, the HTML directory listings will not be saved for any subdirectories inside, as requested; also, the very first time the command runs, there will not be a HTML directory listing saved for myfiles_dir).

However, with the trailing slash - as in http://localhost/shared/myfiles_dir/ - no HTML directory listings will be saved for any directory, including the "root" myfiles_dir, after repeated calls to wget from the same local location.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.