How do I download a set of files according to a pattern?

Question

Here's what I'm trying to do:

1. Download a webpage
2. Find all strings matching a regex in it
3. Prepend and append string constants to them to form fully qualified URLs
4. Download all of them

This should be easy! A bunch of curl and grep strung together with pipes should suffice, but I can't do it!

curl http://example.com/?q=blah | grep -o -P "(?<=alt=\")[^\"]*\"" | what's next?

Also, grep doesn't seem to recognize [^\"].

Things like DownThemAll! are not flexible enough.

Answer 1

I gave up on grep. Hopefully a perl solution is OK.

curl http://example.com/?q=blah | perl -e ' @alt=map { /alt="(.*?)"/ig } ; print "$_\n" foreach(@alt)'

given this input:

afk alt="<I want to find this>" easdfg alt="<I want to find that>" 
adsfsgw wt er ger 
ekfj er  alt="<I want to find this other>" alt="<I want to find this anotherthing>" fgerg
ey 
 ty rth
<img src="file.gif">

it returns this:

<I want to find this>
<I want to find that>
<I want to find this other>
<I want to find this anotherthing>

Answer 2

I notice double quotes are used around the lot, so I guess it's done in cmd.exe (though many linux users use grep, there is the windows version. I welcome any correction to me on this but your example looks to me like it's for the windows implementation one, by virtue of your use of double quotes. I too am using a windows implementation of grep - The gnuwin32 one. (as opposed to the cygwin one for example).

First of all, I would like to correct the error in the line you used with grep(I won't call that an error in your regex, as this is a cmd issue.

Here was your example with your regex with grep that wasn't working.

curl http://example.com/?q=blah | grep -o -P "(?<=alt=\")[^\"]*\"" | what's next?

As you say, [^\"] isn't working

Let's see what exactly is getting passed to grep by cmd (what grep is parsing)

That will require a C program, the C program is included in this question about a quote not working Getting this simple regular expression to match in grep

I will paste the code of the C program here.

You can use this program to see what grep or any windows program receives. (I may be wrong on some technicality here and I welcome a correction if that is so). That said, this does work.

Here is the program that we will use to determine what is going on

#include <stdio.h>

int main(int argc, char *argv[]) {
    int i = 0;
    while (argv[i]) {
        printf("argv[%d] = %s\n", i, argv[i]);
        i++;
    }
    return 0;
}

I've compiled it. w.c , to w.exe

Here is a simple example of what is happening..

Here is an example that works

W:\>w "[^\"]"
argv[0] = w
argv[1] = [^"]

W:\>

You see above, that our program (w) gets 2 parameters, the first is the name of the program (w), the second is [^"]

Now here's an example much smaller that has the same failing as your one, the [^\"] is not working

W:\>w "\"[^\"]"
argv[0] = w
argv[1] = "["]

W:\>

See what grep is getting. I don't know why.. But it looks like when that \" is before the [^\"] and there's double quotes around the whole thing, then the [^\"] doesn't work, we see exactly the result, the [^\"] comes out as ["] We are losing our caret ^

This will preserve our caret, as we can see with the program.

W:\>w "\"[^^\"]"
argv[0] = w
argv[1] = "[^"]

W:\>

Besides that windows issue, there is an issue with your regex you probably want a lookahead for a quote, so you don't match the end quote. You rightly included the lookbehind so as not to match the starting quote.

As an example

W:\>echo blah alt="test" | grep -o -P "(?<=alt=\")[^^\"]*(?=\")"
test

Suppose we have this file called a.a

dsfsdf dfdsf  alt="here" dddd

rrtrtdfddalt="there"dfdfd
alt="df"
tree="dop"

Now we apply

W:\>grep -oP "(?<=alt=\")[^^\"]*(?=\")" a.a
here
there
df

W:\>

So, the regex works and in grep.

W:\>grep -oP "(?<=alt=\")[^^\"]*(?=\")" a.a
here
there
df

W:\>

Now let's suppose I could output that to a file. a.b so a.b now contains those 3 lines. (you can always copy/paste it into a.b)

Now a.b has

here
there
df

let's use sed to add a bit before and after

W:\>sed -r "s#(.*)#http://blah.com/\1.htm#" a.b
http://blah.com/here.htm
http://blah.com/there.htm
http://blah.com/df.htm
W:\>

And we could download all those with wget

W:\>sed -r "s#(.*)#http://blah.com/\1.htm#" a.b >a.c

W:\>type a.c
http://blah.com/here.htm
http://blah.com/there.htm
http://blah.com/df.htm

W:\>wget -i a.c
--2012-07-26 23:21:06--  http://blah.com/here.htm
Resolving blah.com... ^C
W:\>

Answer 3

Checkout wget -r -A.pdf [url] for simple patterns. This will download all pdf from a [url] recursively. This fragment from document is helpful.

‘-A acclist --accept acclist’ ‘-R rejlist --reject rejlist’ Specify comma-separated lists of file name suffixes or patterns to accept or reject (see Types of Files). Note that if any of the wildcard characters, ‘*’, ‘?’, ‘[’ or ‘]’, appear in an element of acclist or rejlist, it will be treated as a pattern, rather than a suffix.