0

I am trying to use httrack to download an entire webarchive from archive.org. The idea is to download only the archive links (as many as possible), but only the links that are really from the archive not from the current website. In other words, I want to download only the links that contain this pattern:

/web/[archive_timestamp]/[website]/*

Here's an example

Here is an archive link: http://web.archive.org/web/20011209181356/http://www.emag.ro:80/

In other to download the links that I need, I am using this command:

httrack http://web.archive.org/web/20011209181356/http://www.emag.ro:80/ -* +*/web/20011209181356/http://www.emag.ro/*

This should mean, to filter all the links (disable all of them), and enable only those that contain /web/20011209181356/http://www.emag.ro/

The command downloads just the homepage, so I guess I'm doing something wrong.

If somebody has an idea of how to get this one done (except from building my own scraper - tried to avoid this in order to save time), even with a different tool that I can use from command line and also works on windows.

1

You were very close - you only needed to add the domain name to your scan rule like this:

httrack http://web.archive.org/web/20011209181356/http://www.emag.ro:80/ -* +web.archive.org/web/20011209181356/http://www.emag.ro/*

Or, to get all archived versions of that domain you could use:

httrack http://web.archive.org/web/20011209181356/http://www.emag.ro:80/ -* +web.archive.org/web/*/http://www.emag.ro/*

But then you'll want to enable traversal throughout the domain by using these command line options (or setting these in the GUI under "Experts Only")

B ... can travel up & down the directory structure
d ... but stay on the same principal domain

See the screenshot below. Is this what you wanted to achieve?

Downloaded pages

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.