Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm trying to download mailing list archives such as

http://cryolist.org/archives/

In particular, I have a page and would like to download every file linked from that page which ends with ".txt.gz". Is there a way to restrict wget to do this? Failing that, an alternate approach?

References

http://www.gnu.org/software/wget/manual/html_node/

share|improve this question
add comment

3 Answers

up vote 3 down vote accepted

Try this:

wget \
    --recursive \
    --no-directories \
    --no-parent \
    --level 1 \
    --accept .txt.gz \
    --execute robots=off \
    http://lists.cryolist.org/pipermail/cryolist-cryolist.org/

Using long options for readability.

share|improve this answer
add comment

From wget --help:

-r,  --recursive          specify recursive download.
-l,  --level=NUMBER       maximum recursion depth (inf or 0 for infinite).
-A,  --accept=LIST        comma-separated list of accepted extensions.

Keeping in mind that the files are linked to in the iFrame lists.cryolist.org/pipermail/cryolist-cryolist.org, the following command will produce the desired results:

wget -r -l1 -A "txt.gz" lists.cryolist.org/pipermail/cryolist-cryolist.org

Switches

  • The -r switch makes download other resources that are linked to.

  • The -l1 switch limits this to one level, i.e., if page1 links to page2 and page2 links to unwanted.txt.gz, this file will not be downloaded.

share|improve this answer
add comment

Write a bash script, Store all the months in an array, loop though them inserting them in to the wget command at the correct position

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.