I would like to crawl my website and generate a list of internal and outbound links, and what their final destination is and HTTP Code (at least for internal links). How can I do that?

Please note: I don't care about the content of the page except to find links in it.

  • Software recommendations are not done through here. Since I like your picture go have a look here wummel.github.io/linkchecker – Matthew Williams Feb 10 '14 at 16:29
  • Already tried linkchecker, I can't get it to spitout a list of all the links on the page. I just checks that they're valid and reports errors. – Malfist Feb 10 '14 at 16:30
  • Please show what your research tells you and what you've already tried. – slhck Feb 10 '14 at 16:35

The following might be considered:

http://www.httrack.com - Allthough its built to download websites, but you can just generate a report with all link.

http://scrapy.org/download/ - Its most suitable (I think) for your needs.

I have personally used HtTrack on Windows. Its available for Linux as well. If you need additional details, add a comment.

| improve this answer | |
  • While scrappy would work, which is why I accepted this answer. However, I wound up using a tool that generates sitemaps. – Malfist Feb 10 '14 at 20:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.