Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to retrieve a whole website map, in a format like :

I need it to be linked-based (no file or dir brute-force), like :

parse homepage -> retrieve all links -> explore them -> retrieve links, ...

And I also need the ability to detect if a page is a "template" to not retrieve all of the "child-pages". For example if the following links are found :

I need to get only once the http://example.org/product/viewproduct

I've looked into HTTtracks, wget (with spider-option), but nothing conclusive so far.

The soft/tool should be downloadable, and I prefer if it runs on Linux. It can be written in any language.

Thanks

share|improve this question

closed as off topic by Indrek, Diogo, soandos, Nifle, Ƭᴇcʜιᴇ007 Sep 30 '12 at 17:46

Questions on Super User are expected to relate to computer software or computer hardware within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

5 Answers 5

up vote 3 down vote accepted

After a lot of research no tool has satisfied me, therefore I'm coding my own using http://scrapy.org/doc/

share|improve this answer

I personnaly use Kapow Katalyst, but I guess it's out of your budget. If not, it's probably the most intuitive software to create spiders, and much more if you need.

share|improve this answer
    
Thanks, I didn't know about this one. I'll take a look, although I don't have budget for this at this time. –  ack__ Sep 18 '12 at 8:23

Here is an example of one made in python:

(Taken from http://theanti9.wordpress.com/2009/02/14/python-web-crawler-in-less-than-50-lines/ )

Also on that website there is a link to a github project http://github.com/theanti9/PyCrawler that is a more robust version the person made.

import sys
import re
import urllib2
import urlparse
tocrawl = set(["http://www.facebook.com/"])
crawled = set([])
keywordregex = re.compile('<meta\sname=["\']keywords["\']\scontent=["\'](.*?)["\']\s/>')
linkregex = re.compile('<a\s*href=[\'|"](.*?)[\'"].*?>')

while 1:
    try:
        crawling = tocrawl.pop()
        print crawling
    except KeyError:
        raise StopIteration
    url = urlparse.urlparse(crawling)
    try:
        response = urllib2.urlopen(crawling)
    except:
        continue
    msg = response.read()
    startPos = msg.find('<title>')
    if startPos != -1:
        endPos = msg.find('</title>', startPos+7)
        if endPos != -1:
            title = msg[startPos+7:endPos]
            print title
    keywordlist = keywordregex.findall(msg)
    if len(keywordlist) > 0:
        keywordlist = keywordlist[0]
        keywordlist = keywordlist.split(", ")
        print keywordlist
    links = linkregex.findall(msg)
    crawled.add(crawling)
    for link in (links.pop(0) for _ in xrange(len(links))):
        if link.startswith('/'):
            link = 'http://' + url[1] + link
        elif link.startswith('#'):
            link = 'http://' + url[1] + url[2] + link
        elif not link.startswith('http'):
            link = 'http://' + url[1] + '/' + link
        if link not in crawled:
            tocrawl.add(link)
share|improve this answer

Technically speaking there is no Full-proof way of extracting the directory structure of a website.
This is because HTTP is not a network file system. The only thing you can do, with HTTP, is follow the links from the starting page. Furthermore, there's nothing that requires the starting page to have links only to its immediate subdirectory. A top level index.html page may, for example, have a direct link to "foo/baz/blah.html", deep in some subdirectory.

Edit:

  • To generate basic site maps, Some online tools are there commonly known as Sitemap Generator. One such tool is this, it gives sitemap in XML.

  • If you are comfortable with programming then you can write your own web-spider, with specific set of rules of a particular site.

share|improve this answer
    
Indeed I'm looking for a follow-link style spider. No problem with sites not having links only to sub-dirs, the soft can later trim the content found and organize it in tree-view. I don't want to rely on XML sitemaps as they don't present all the site's content. And as to program my own spider, this is something much more complicated than it looks (see various threads on stackoverflow), and it takes a huge lot of time. –  ack__ Sep 4 '12 at 10:39

(Win)HTTrack does a very decent job.

It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer.

share|improve this answer
1  
While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. –  BBlake Sep 12 '12 at 20:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.