Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

If you have used Heritrix Web Crawler, I'd really appreciate your help.

3 questions:

  1. An arc file probably contains source codes of MANY pages in there. How do I figure out which is which?
  2. How do I interpret the .arc.gz files? I opened them in VIM and realized there were HTML code + junk (which I can't even parse using Python SGMLParser because of the junk).
  3. Is it recommended to compress? (.gz)

Basically, I have no idea what .ARC files are and what I can do with them. I'm used to using URLLIB2 to download and parse HTML manually.

share|improve this question

migrated from stackoverflow.com Oct 16 '09 at 19:42

This question came from our site for professional and enthusiast programmers.

1 Answer 1

up vote 3 down vote accepted

Her's a link to download ArcReader, and an explanation: http://crawler.archive.org/articles/developer%5Fmanual/arcs.html.

I Googled for reading arc files and this was the first link.

First you need to unzip the files (they are gzipped, hence the .gz extension.). Then you can read the ARC file.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.