Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Here's what I'm trying to do:

  1. Download a webpage
  2. Find all strings matching a regex in it
  3. Prepend and append string constants to them to form fully qualified URLs
  4. Download all of them

This should be easy! A bunch of curl and grep strung together with pipes should suffice, but I can't do it!

curl http://example.com/?q=blah | grep -o -P "(?<=alt=\")[^\"]*\"" | what's next?

Also, grep doesn't seem to recognize [^\"].

Things like DownThemAll! are not flexible enough.

share|improve this question
    
Don't use regexps to parse html –  Nifle Jul 26 '12 at 10:56
1  
I didn't say the webpage contains HTML. I didn't say I want to parse it. –  Ansis Malins Jul 26 '12 at 10:58
    
The link was more or less a joke. If you have a very good idea how the data/html looks it's usually possible to make it work with regexps. You just can't make it robust (you'll have to expect a failure now and then). –  Nifle Jul 26 '12 at 11:05
    
I have some problem with understanding what kind of strings you want to capture. Could you give a few examples? –  Nifle Jul 26 '12 at 11:05
    
I want to find the contents of all alt attributes. alt="<I want to find this>" They contain filenames. –  Ansis Malins Jul 26 '12 at 11:15

3 Answers 3

I gave up on grep. Hopefully a perl solution is OK.

curl http://example.com/?q=blah | perl -e ' @alt=map { /alt="(.*?)"/ig } ; print "$_\n" foreach(@alt)'

given this input:

afk alt="<I want to find this>" easdfg alt="<I want to find that>" 
adsfsgw wt er ger 
ekfj er  alt="<I want to find this other>" alt="<I want to find this anotherthing>" fgerg
ey 
 ty rth
<img src="file.gif">

it returns this:

<I want to find this>
<I want to find that>
<I want to find this other>
<I want to find this anotherthing>
share|improve this answer

I notice double quotes are used around the lot, so I guess it's done in cmd.exe (though many linux users use grep, there is the windows version. I welcome any correction to me on this but your example looks to me like it's for the windows implementation one, by virtue of your use of double quotes. I too am using a windows implementation of grep - The gnuwin32 one. (as opposed to the cygwin one for example).

First of all, I would like to correct the error in the line you used with grep(I won't call that an error in your regex, as this is a cmd issue.

Here was your example with your regex with grep that wasn't working.

curl http://example.com/?q=blah | grep -o -P "(?<=alt=\")[^\"]*\"" | what's next?

As you say, [^\"] isn't working

Let's see what exactly is getting passed to grep by cmd (what grep is parsing)

That will require a C program, the C program is included in this question about a quote not working Getting this simple regular expression to match in grep

I will paste the code of the C program here.

You can use this program to see what grep or any windows program receives. (I may be wrong on some technicality here and I welcome a correction if that is so). That said, this does work.

Here is the program that we will use to determine what is going on

#include <stdio.h>

int main(int argc, char *argv[]) {
    int i = 0;
    while (argv[i]) {
        printf("argv[%d] = %s\n", i, argv[i]);
        i++;
    }
    return 0;
}

I've compiled it. w.c , to w.exe

Here is a simple example of what is happening..

Here is an example that works

W:\>w "[^\"]"
argv[0] = w
argv[1] = [^"]

W:\>

You see above, that our program (w) gets 2 parameters, the first is the name of the program (w), the second is [^"]

Now here's an example much smaller that has the same failing as your one, the [^\"] is not working

W:\>w "\"[^\"]"
argv[0] = w
argv[1] = "["]

W:\>

See what grep is getting. I don't know why.. But it looks like when that \" is before the [^\"] and there's double quotes around the whole thing, then the [^\"] doesn't work, we see exactly the result, the [^\"] comes out as ["] We are losing our caret ^

This will preserve our caret, as we can see with the program.

W:\>w "\"[^^\"]"
argv[0] = w
argv[1] = "[^"]

W:\>

Besides that windows issue, there is an issue with your regex you probably want a lookahead for a quote, so you don't match the end quote. You rightly included the lookbehind so as not to match the starting quote.

As an example

W:\>echo blah alt="test" | grep -o -P "(?<=alt=\")[^^\"]*(?=\")"
test

Suppose we have this file called a.a

dsfsdf dfdsf  alt="here" dddd

rrtrtdfddalt="there"dfdfd
alt="df"
tree="dop"

Now we apply

W:\>grep -oP "(?<=alt=\")[^^\"]*(?=\")" a.a
here
there
df

W:\>

So, the regex works and in grep.

W:\>grep -oP "(?<=alt=\")[^^\"]*(?=\")" a.a
here
there
df

W:\>

Now let's suppose I could output that to a file. a.b so a.b now contains those 3 lines. (you can always copy/paste it into a.b)

Now a.b has

here
there
df

let's use sed to add a bit before and after

W:\>sed -r "s#(.*)#http://blah.com/\1.htm#" a.b
http://blah.com/here.htm
http://blah.com/there.htm
http://blah.com/df.htm
W:\>

And we could download all those with wget

W:\>sed -r "s#(.*)#http://blah.com/\1.htm#" a.b >a.c

W:\>type a.c
http://blah.com/here.htm
http://blah.com/there.htm
http://blah.com/df.htm

W:\>wget -i a.c
--2012-07-26 23:21:06--  http://blah.com/here.htm
Resolving blah.com... ^C
W:\>
share|improve this answer

Checkout wget -r -A.pdf [url] for simple patterns. This will download all pdf from a [url] recursively. This fragment from document is helpful.

‘-A acclist --accept acclist’ ‘-R rejlist --reject rejlist’ Specify comma-separated lists of file name suffixes or patterns to accept or reject (see Types of Files). Note that if any of the wildcard characters, ‘*’, ‘?’, ‘[’ or ‘]’, appear in an element of acclist or rejlist, it will be treated as a pattern, rather than a suffix.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.