Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

How would I download a list of files from a file server like this one http://www.apache.org/dist/httpd/binaries/ ?

I suppose I could use wget but then it tries to get all the links and the html file as well. Is there a better tool to accomplish this?

share|improve this question
just to clarify your question: you just want the list of files which could be downloaded from the server, not the files itself (yet)? –  akira Sep 26 '09 at 4:57
In what way is a command like ` wget --no-verbose --spider --no-directories --recursive --level=2 apache.org/dist/httpd/binaries` not working for you? If you could be more specific that might help –  DaveParillo Sep 26 '09 at 5:02

2 Answers 2

You can specify what file extensions wget will download when crawling pages:

wget -r -A zip,rpm,tar.gz www.site.com/startpage.html

this will perform a recursive search and only download files with the .zip, .rpm, and .tar.gz extensions.

share|improve this answer

supposing you really just want a list of the files on the server without fetching them (yet):

%> wget -r -np --spider http://www.apache.org/dist/httpd/binaries/ 2>&1 | awk -f filter.awk | uniq

while 'filter.awk' looks like this

/^--.*--  http:\/\/.*[^\/]$/ { u=$3; }
/^Length: [[:digit:]]+/ { print u; }

then you possibly have to filter out some entries like

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.