Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Not really a question this one - but alternative answers are certainly welcome! :)

Sometimes I need to download files listed in an Apache directory listing, say as in:

... in a given directory on my computer, and I'd like to use command-line tools for that.

As far as wget is concerned, actually there are some switches that should be in place - noting that wget tends to either download single items - or to reconstruct the server folders locally!

As I just spent some half an hour figuring out what those proper switches are (to copy the remote files locally as through, say, FTP), I'd like to document them here; so the above link would be downloaded with:

wget -nd -r -l 1 http://www.gnu-darwin.org/www001/src/ports/net/samba-libsmbclient/work/samba-3.0.28/source/libsmb/

... where:

   -nd
   --no-directories
       Do not create a hierarchy of directories when retrieving
       recursively.  With this option turned on, all files will get saved
       to the current directory, without clobbering [...]

   -r
   --recursive
       Turn on recursive retrieving.

   -l depth
   --level=depth
       Specify recursion maximum depth level depth.  The default maximum
       depth is 5.

It would be nice to hear alternatives to the above command - maybe using different set of switches for wget - or maybe using curl or other packages...

share|improve this question
    
Also related: unix.stackexchange.com/questions/36798/… – sdaau Oct 8 '14 at 9:18

See also -nH a.k.a --no-host-directories and --cut-dirs options. I also frequently use --accept/-A and --reject/-R.

share|improve this answer
    
Many thanks for these options, @jw013 - great to have a quick reference for them! Cheers! – sdaau Sep 19 '11 at 19:24

After some time, thanks to the pointers by @jw013 and in Sync with a Directory Listing of An Apache Server - Unix & Linux Stack Exchange, I finally found my "ultimate" wget command.

Assume you have a tree of files and directories in a directory called myfiles_dir, and let that directory be accessible through an Apache directory listing on http://localhost/shared/myfiles_dir. Then, you can sync directory locally with:

wget -r -N --no-parent --reject '*index.html*' -nH --cut-dirs=1 http://localhost/shared/myfiles_dir/

... where:

-r,  --recursive              specify recursive download.
-N,  --timestamping           don't re-retrieve files unless newer than
                              local.
-np, --no-parent              don't ascend to the parent directory.
-R,  --reject=LIST            comma-separated list of rejected extensions.
-nH, --no-host-directories    don't create host directories.
     --cut-dirs=NUMBER        ignore NUMBER remote directory components.

This will download the contents into a subdirectory myfiles_dir created in the directory wget was called from (the working directory), without any residual index.html files.

Note that the trailing slash / after the http link/address is extremely important: if it is not there - as in http://localhost/shared/myfiles_dir - upon repeated calls of wget from the same local working directory, the HTML of the directory listing will be saved as multiple copies e.g. myfiles_dir.1, myfiles_dir.2 etc, in spite of any switches (although, the HTML directory listings will not be saved for any subdirectories inside, as requested; also, the very first time the command runs, there will not be a HTML directory listing saved for myfiles_dir).

However, with the trailing slash - as in http://localhost/shared/myfiles_dir/ - no HTML directory listings will be saved for any directory, including the "root" myfiles_dir, after repeated calls to wget from the same local location.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .