Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to download all the PDF files present on a site. Trouble is, they aren't listed on any one page, so I need something (a program? a framework?) to crawl the site and download the files, or at least get a list of the files. I tried WinHTTrack, but I couldn't get it to work. DownThemAll for Firefox does not crawl multiple pages or entire sites. I know that there is a solution out there, as I couldn't have possibly been the first person to be presented with this problem. What would you recommend?

share|improve this question

migrated from Jul 7 '10 at 14:46

This question came from our site for professional and enthusiast programmers.


wget -r -A pdf
share|improve this answer
what's up Roque! Bayern München fan? – Pablo Santa Cruz Jul 7 '10 at 11:59

Google has an option to return only files of a certain type. Combine this with the "site" option and you have your "crawler".


share|improve this answer

Use some webcrawling library, eg. in ruby

share|improve this answer

If there are no links to PDF files, a crawler won't help and you basically only have two choices:

  1. Get the list from somewhere else (ask the site's Web Master for a list)
  2. Get the list from WebSite's directory listing. Although, if they have disabled this option on their web server, you won't be able to use it.
share|improve this answer
Finally, we meet - hi Pablo ;) – miku Jul 7 '10 at 11:59

You must log in to answer this question.