up vote 1 down vote favorite
share [g+] share [fb]

I need to download all the PDF files present on a site. Trouble is, they aren't listed on any one page, so I need something (a program? a framework?) to crawl the site and download the files, or at least get a list of the files. I tried WinHTTrack, but I couldn't get it to work. DownThemAll for Firefox does not crawl multiple pages or entire sites. I know that there is a solution out there, as I couldn't have possibly been the first person to be presented with this problem. What would you recommend?

link|improve this question
feedback

3 Answers

up vote 3 down vote

Is the site indexed by Google? You could just search site:www.example.com filetype:pdf and it would return all of the PDF files indexed on that site.

Example: site:en.wikipedia.org filetype:pdf

Another way to do it would be by using wget, which is a *nix command line tool, but is available for windows. By using recursive mode and specifying to only save pdf files, it could traverse a site and find all the PDFs.

wget.exe -r -A pdf -np http://www.example.com/ will recursively download all PDF files from example.com, and will not try to go into a parent directory (only relevant if you want to start in a specific subdirectory).

I haven't actually tested using WGET with just a specific filetype before, but I've done recursive downloads before and it's worked fine, so you might want to give it a try.

link|improve this answer
This is great, I would love to use wget. The trouble is, the site in question is a Drupal based site where the links to the PDFs end in numbers instead of .pdf. So, this wget command does not work unless there is a way to tell wget to manually check the filetype of all inks on the site. Maybe something like --html-extension but for PDFs? I looked over the wget docs at linux.die.net/man/1/wget and I didn't see anything like that. Also, the site is poorly indexed by Google, so I'd like to do the crawling myself to make sure I get everything. – user43112 Jul 25 '10 at 2:59
feedback
up vote 1 down vote

I can advice you to use jsoup, the Java HTML Parser if you're familiar with Java or BeautifulSoup if you're familiar with Python.

Write recursive method which do next:

  1. find all links on page which have tag "a" and attribute "href" get it value and compare link with domain name.
  2. chek extension of link pdf or not
  3. if this link have pdf extension download it or get content if this link is match with domain and do the same operation again.

Something like that.

link|improve this answer
Welcome to SuperUser. There's nothing wrong with posting an additional answer on an old question if you are able to provide further insights. – Oliver Salzburg May 4 at 12:58
thnx, Oliver. so i just write same script before if i found them i can attache it if so, but sometimes people want to solve the problems by himself ;) with little advice – Ishikawa Yoshi May 4 at 13:14
feedback
up vote 0 down vote

This is not the easiest solution, but it's my favorite.

Write your own spider! Have it traverse the site, parse pages, and look for files you want. There are plenty of interesting challenges that come with writing a spider, and I found it a lot of fun.

While this may not be what archer1742 was looking for, I'm sure someone (perhaps archer1742 as well) would like this solution.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.