Limiting files with wget

Question

I'm trying to download mailing list archives such as

http://cryolist.org/archives/

In particular, I have a page and would like to download every file linked from that page which ends with ".txt.gz". Is there a way to restrict wget to do this? Failing that, an alternate approach?

References

http://www.gnu.org/software/wget/manual/html_node/

Answer 1

Try this:

wget \
    --recursive \
    --no-directories \
    --no-parent \
    --level 1 \
    --accept .txt.gz \
    --execute robots=off \
    http://lists.cryolist.org/pipermail/cryolist-cryolist.org/

Using long options for readability.

Answer 2

From wget --help:

-r,  --recursive          specify recursive download.
-l,  --level=NUMBER       maximum recursion depth (inf or 0 for infinite).
-A,  --accept=LIST        comma-separated list of accepted extensions.

Keeping in mind that the files are linked to in the iFrame lists.cryolist.org/pipermail/cryolist-cryolist.org, the following command will produce the desired results:

wget -r -l1 -A "txt.gz" lists.cryolist.org/pipermail/cryolist-cryolist.org

Switches

• The -r switch makes download other resources that are linked to.

• The -l1 switch limits this to one level, i.e., if page1 links to page2 and page2 links to unwanted.txt.gz, this file will not be downloaded.

Answer 3

Write a bash script, Store all the months in an array, loop though them inserting them in to the wget command at the correct position