Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am seeking a tool to space-efficiently archive a blog that is changing every day or even two or three times a day. I don't mean that individual blog posts change - not regularly anyway - I just mean that new blog entries are added and older entries are shifted down the front page. One problem I see is that it will be inefficient to archive the same blog entry multiple times. Revisions to the same entry should be archived, ideally, but the original need not be since the revision is likely due to an improvement or correction.

It is a blog with text and static images. A linux solution is preferred.

share|improve this question

One solution is to store it in a Git repository.

Since Git uses content-based addressing, unchanged files take up negligible additional space in the repository. Revisions also take up little space because it stores diffs. Initially the blobs are stored individually compressed, but Git periodically combines files into packs, which are compressed more effectively. You can also manually invoke that functionality using git gc.

A simple way to fetch the website data is to use wget --mirror. Alternatively, look to see whether the blog site provides an XML API (which would be more space-efficient by avoiding archiving boilerplate HTML). You want to download the pages into the current working tree.

Then, after the download finishes, add and commit everything to the git repository. Hence each commit represents a snapshot in time.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .