I have a html-page url and I want to grep it. How can I do it by wget someArgs | grep keyword?
My first idea was wget -q -O - url | grep keyword, but wget's output bypass grep and arise on the terminal in its original form.
asked Jun 1, 2012 at 19:34
-
grep selects lines delimited by (e.g.) carriage return and linefeed characters, an HTML response doesn't have lines it has text with markup like <br> or <p> so the whole web-page could look like one line to grep– RedGrittyBrickJun 1, 2012 at 19:44
-
1@RedGrittyBrick The OP's command works flawlessly for me.– slhckJun 1, 2012 at 19:47
Add a comment
|
5 Answers
The easiest way is to use curl with the option -s for silent:
curl -s http://somepage.com | grep whatever
-
-
@Dennis Try
curlinghttp://superuser.com/questions/431581. For whatever reason I tested it with this particular URL and got no output. Dunno what I'm missing.– slhckJun 1, 2012 at 21:45 -
1@slhck: Curl doesn't follow redirects by default. It does with the
-Lswitch.– DennisMar 31, 2013 at 20:15 -
@Dennis Didn't know what you were talking about without seeing the deleted comments – but yeah, that makes sense. Thanks for clearing it up.– slhckMar 31, 2013 at 20:27
-
1Question asks for wget. Not curl. This won't work with multiple redirects and -L option. Oct 7, 2016 at 23:07
Keeping this around for the sake of completeness.
Your example should actually work. The syntax is correct, and here's a screencast I just took demonstrating it, with a good old GNU wget 1.13.4.
wget -q some-url -O - | grep something
So assume your pattern is wrong and grep will just output everything it got.
-
It could also be a typo in the URL. With
-q, there is no error message.– DennisJun 2, 2012 at 0:44
If you are looking to grep or pipe headers, they are standard directed to stderr so you need to redirect them. Eg:
wget -O - http://example.com/page.php > /dev/null 2>&1 | grep HTTP
This bug was in v1.12.1 fixed in another version. Currently I use v1.15 and it works as expected.
The wget writes its output to stderr not to stdout, so one needs to redirect the stderr to stdout:
wget -q -O - url 2&>1 | grep keyword
