up vote 0 down vote favorite
share [g+] share [fb]

I want to pass curl the output from awk

./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | curl OUTPUT_FROM_AWK | grep myString

How can I achieve this?!

link|improve this question
50% accept rate
feedback

2 Answers

up vote 1 down vote accepted

Use xargs.

xargs utility [argument ...]

The xargs utility reads space, tab, newline and end-of-file delimited strings from the standard input and executes utility with the strings as arguments.

There are more parameters and options than in this shortened form, of course.

A general example using curl:

$ echo "http://www.google.com" | xargs curl
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.de/">here</A>.
</BODY></HTML>

In your specific case, it'd look similar to the following:

./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | xargs curl | grep myString
link|improve this answer
Great! However in my case this doesn't work. If piping to "xargs echo" it also doesn't work. I guess output of jspider is too fast or sth.. – Stephan Kristyn Dec 2 '11 at 17:13
In that case, consider adding to your question or creating a new question dealing with your specific problem. I don't have this specific issue, e.g. curl -s "http://superuser.com" | grep -E 'href="http://.*stackexchange\.com' | sed 's|^.*<a href="http://\([^"]*\)">.*$|http://\1|g' | grep -v "<" | xargs curl -s | grep "<title>" works fine for me. (Yeah I know the code's extremely hacky). – Daniel Beck Dec 2 '11 at 18:10
Oh my.. I have to de-cypher that sed part in order to understand what you are doing, still it isn't working with jspider. I will start using perl now. – Stephan Kristyn Dec 5 '11 at 9:39
@StephanKristyn I extract URLs from hyperlinks, and since it doesn't work well enough, using grep -v I then remove all remaining lines with HTML tag brackets. Just a simple example that shows the approach works in general. – Daniel Beck Dec 5 '11 at 9:51
feedback
up vote 0 down vote

Try this, it's untested but should work.

for a in $(./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}'); do curl $a | grep myString; done
link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.