Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

A large biological research project has chosen to make its archive available via https here:

Unfortunately, it appears that there is no manifest of the contents of these directories, so I simply want to build one. I'd like to grab the filenames for the entire directory tree. Are there any suggestions for how to do this? I can write something up in perl/python/R/etc. to scrape the index.html files recursively, but I thought there might be some incantation with wget that can get me the filenames but I have not found that yet.

share|improve this question
up vote 3 down vote accepted

I actually had the exact same problem. Both of these solutions didn't work for me. However, this did:

Install lftp, then do

du -a > manifest.txt

and that'll give you all the directories and file names.

share|improve this answer

Opposed to the FTP protocol, HTTP does not know the concept of a directory listing. Thus, wget can only look for links and follow them according to certain rules the user defines.

That being said, if you absolutely want it, you can abuse wgets debug mode to gather a list of the links it encounters when analyzing the HTML pages. It sure ain't no beauty, but here goes:

wget -d -r -np -N --spider -e robots=off --no-check-certificate \ \
  2>&1 | grep " -> " | grep -Ev "\/\?C=" | sed "s/.* -> //"

Some sidenotes:

  • This will produce a list which still contains duplicates (of directories), so you need to redirect the output to a file and use uniq for a pruned list.
  • --spider causes wget not to download anything, but it still will do a HTTP HEAD request on each of the files it deems to enqueue. This will cause a lot more traffic than is actually needed/intended and cause the whole thing to be quite slow.
  • -e robots=off is needed to ignore a robots.txt file which may cause wget to not start searching (which is the case for the server you gave in your question).
  • If you have wget 1.14 or newer, you can use --reject-regex="\?C=" to reduce the number of needless requests (for those "sort-by" links already mentioned by @slm). This also eliminates the need for the grep -Ev "\/\?C=" step afterwards.
share|improve this answer

I thought there would be a way to do this easily with wget/curl too but couldn't get anything to work either. You can use this Ruby gem, anemone, to do it fairly easily though.

Installing anemone gem

% gem install anemone
Fetching: robotex-1.0.0.gem (100%)
Fetching: anemone-0.7.2.gem (100%)
Successfully installed robotex-1.0.0
Successfully installed anemone-0.7.2
2 gems installed
Installing ri documentation for robotex-1.0.0...
Installing ri documentation for anemone-0.7.2...
Installing RDoc documentation for robotex-1.0.0...
Installing RDoc documentation for anemone-0.7.2...

Sample anemone script

#! /usr/bin/env ruby
require 'anemone'

Anemone.crawl("") do |anemone|
  anemone.on_every_page do |page|
      puts page.url

Example run

% ./anemone.rb | grep -v '?C='

NOTE: The bit grep -v '?C=' is filtering the boilerplate headers that Apache is generating via its Indexing directive, i.e.:

IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable

    ss of apache rendered column sorter

These allow you to sort the pages by the different columns (Name, Create Date, etc.). These show up as pages and I'm just filtering them out of the output.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .