Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If you have used Heritrix Web Crawler, I'd really appreciate your help.

3 questions:

  1. An arc file probably contains source codes of MANY pages in there. How do I figure out which is which?
  2. How do I interpret the .arc.gz files? I opened them in VIM and realized there were HTML code + junk (which I can't even parse using Python SGMLParser because of the junk).
  3. Is it recommended to compress? (.gz)

Basically, I have no idea what .ARC files are and what I can do with them. I'm used to using URLLIB2 to download and parse HTML manually.

share|improve this question

migrated from stackoverflow.com Oct 16 '09 at 19:42

This question came from our site for professional and enthusiast programmers.

up vote 3 down vote accepted

Her's a link to download ArcReader, and an explanation: http://crawler.archive.org/articles/developer%5Fmanual/arcs.html.

I Googled for reading arc files and this was the first link.

First you need to unzip the files (they are gzipped, hence the .gz extension.). Then you can read the ARC file.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .