What I mean is to download each page available from the Wayback Machine over a specified time period and interval. For example, I want to download each page available from each day from nature.com from January of 2012 to December of 2012. (Not precisely what I want to do, but it's close enough -- and provides a good example.)

wget won't work due to the unique nature of how the Wayback machine works, unfortunately.

Tools like Wayback Machine downloader only download the most recent version of the page, it seems.

Interacting with the IA API seems like a viable route, but I'm not sure how that would work.


  • You would definitely need to write a script for this. Maybe cURL?
    – pulsejet
    Mar 15, 2017 at 14:50
  • I think it'd be possible to write a script and lean on cURL, but I'm unfamiliar with the Memento API that the Internet Archive uses, and don't' think I've seen it used in this way. Mar 16, 2017 at 14:38
  • I need to a) Do multiple sites at once, b) grab a snapshot of each site over a long interval (say, 1998 to 2001), and c) be able to specify how many snapshots I want to take over that interval. Mar 16, 2017 at 14:52
  • Possible duplicate: superuser.com/questions/828907/…
    – pulsejet
    Mar 16, 2017 at 18:17
  • Same problem. They just want one page, it seems -- the documentation for the WB Machine downloader is vague whether it works over an interval like that, or not. Mar 16, 2017 at 22:02

3 Answers 3


The way wayback URLs are formatted are as follows:


Here BASEURL is usually http://web.archive.org/web (I say usually as I am unsure if it is the only BASEURL)

TARGET is self explanatory (in your case http://nature.com, or some similar URL)

TIMESTAMP is YYYYmmddHHMMss when the capture was made (in UTC):

  • YYYY: Year
  • mm: Month (2 digit - 01 to 12)
  • dd: Day of month (2 digit - 01 to 31)
  • HH: Hour (2 digit - 00 to 23)
  • MM: Minute (2 digit - 00 to 59)
  • ss: Second (2 digit - 00 to 59)

In case you request a capture time that doesn't exist, the wayback machine redirects to the closest capture for that URL, whether in the future or the past.

You can use that feature to get each daily URL using curl -I (HTTP HEAD) to get the set of URLs:

START=1325419200 # Jan 1 2012 12:00:00 UTC (Noon) 
END=1356998400 # Tue Jan  1 00:00:00 UTC 2013
if uname -s |grep -q 'Darwin' ; then
    DATECMD="date -u '+%Y%m%d%H%M%S' -r "
elif uname -s |grep -q 'Linux'; then
    DATECMD="date -u +%Y%m%d%H%M%S -d @"

while [[ $START -lt $END ]]; do
    REDIRECT="$(curl -sI "$BASEURL/$TIMESTAMP/$TARGET" |awk '/^Location/ {print $2}')"
    if [[ -z "$REDIRECT" ]]; then
        echo $REDIRECT
    START=$((START + 86400)) # add 24 hours

This gets you the URLs that are closest to noon on each day of 2012. Just remove the duplicates, and, and download the pages.

Note: The Script above can probably be greatly improved to jump forward in case the REDIRECT is for a URL more than 1 day in the future, but then it requires deconstructing the returned URL, and adjusting START to the correct date value.

  • This is great, why? because we have facts and proof of when somebody archived content and web.archive.org has removed archived content in the past. This script above would save archived content. Awesome. Mar 23, 2017 at 13:10
  • It does only download the main file though, and not related js, html, css.
    – Purpose
    Mar 1, 2020 at 20:17
  • Actually this script doesn't download anything: It just shows the closest URL. You can plug that URL into wget or any other page-getter with the correct options (notice the -I option to curl).
    – Samveen
    Mar 5, 2020 at 11:42

There is a ruby gem on Github: https://github.com/hartator/wayback-machine-downloader

  • This is awesome. Mar 21, 2017 at 13:08

There is also a nice Python package for those more inclined to Python:


It's as simple as:

waybackpack dol.gov -d ~/Downloads/dol-wayback --to-date 1996

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .