Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to download mailing list archives such as

http://cryolist.org/archives/

In particular, I have a page and would like to download every file linked from that page which ends with ".txt.gz". Is there a way to restrict wget to do this? Failing that, an alternate approach?

References

http://www.gnu.org/software/wget/manual/html_node/

share|improve this question
up vote 3 down vote accepted

Try this:

wget \
    --recursive \
    --no-directories \
    --no-parent \
    --level 1 \
    --accept .txt.gz \
    --execute robots=off \
    http://lists.cryolist.org/pipermail/cryolist-cryolist.org/

Using long options for readability.

share|improve this answer

From wget --help:

-r,  --recursive          specify recursive download.
-l,  --level=NUMBER       maximum recursion depth (inf or 0 for infinite).
-A,  --accept=LIST        comma-separated list of accepted extensions.

Keeping in mind that the files are linked to in the iFrame lists.cryolist.org/pipermail/cryolist-cryolist.org, the following command will produce the desired results:

wget -r -l1 -A "txt.gz" lists.cryolist.org/pipermail/cryolist-cryolist.org

Switches

  • The -r switch makes download other resources that are linked to.

  • The -l1 switch limits this to one level, i.e., if page1 links to page2 and page2 links to unwanted.txt.gz, this file will not be downloaded.

share|improve this answer

Write a bash script, Store all the months in an array, loop though them inserting them in to the wget command at the correct position

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .