Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The problem is this, I have a file with a list of URLs, say links.txt:

What I'm trying to do is let wget know that I don't want it to get png like so:

$ wget -R png -i links.txt

But this has no effect and wget still gets png files along with others. Piping links.txt through grep is not an option as in actual file the links are in this form that then gets resolved/redirected into something like

So the question is, how do I reject/exclude certain files with wget?

share|improve this question
Is order important? Try wget -i links.txt -R png – Kusalananda Nov 3 '11 at 10:22
Nope. Order is not important. – grimgav Nov 3 '11 at 11:08
up vote 1 down vote accepted

Wget, or at least the version I have appears poorly equipped to do this, the --server-response option still seems to download the file. If wget isn't critical, then curl may be a better option.

The solution to this type of problem involves looking at the Content-Type returned by the server. For example:

curl -I

writes the something like the following on stdout
HTTP/1.1 200 OK
Server: nginx admin
Date: Thu, 03 Nov 2011 09:22:55 GMT
Content-Type: image/png
Content-Length: 35765
Last-Modified: Wed, 13 Apr 2011 05:19:19 GMT
Connection: keep-alive
Vary: Accept-Encoding
Expires: Thu, 10 Nov 2011 09:22:55 GMT
Cache-Control: max-age=604800
X-Cache: HIT from Backend
Accept-Ranges: bytes

Filtering that result with grep allows you to test for acceptable mime types. You can then generalize the approach to check for valid mime types for lists of files. Tidying this up and putting it into a shell script:

# in


function mimefilter {
    curl -sI $URL | egrep ${PATTERN} > /dev/null 2>&1
    if [ "$?" -eq "0" ] ; then
        wget $URL

    while read line
        mimefilter $line $PATTERN

Which you would call in this way:

mimechecker links.txt 'image/png'
share|improve this answer
Great idea, thank you for sharing and answering my question. That really helped. – grimgav Nov 3 '11 at 11:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .