Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I want to pass curl the output from awk

./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | curl OUTPUT_FROM_AWK | grep myString

How can I achieve this?!

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

Use xargs.

xargs utility [argument ...]

The xargs utility reads space, tab, newline and end-of-file delimited strings from the standard input and executes utility with the strings as arguments.

There are more parameters and options than in this shortened form, of course.


A general example using curl:

$ echo "http://www.google.com" | xargs curl
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.de/">here</A>.
</BODY></HTML>

In your specific case, it'd look similar to the following:

./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | xargs curl | grep myString
share|improve this answer
    
Great! However in my case this doesn't work. If piping to "xargs echo" it also doesn't work. I guess output of jspider is too fast or sth.. –  Achmed Durangi Dec 2 '11 at 17:13
    
In that case, consider adding to your question or creating a new question dealing with your specific problem. I don't have this specific issue, e.g. curl -s "http://superuser.com" | grep -E 'href="http://.*stackexchange\.com' | sed 's|^.*<a href="http://\([^"]*\)">.*$|http://\1|g' | grep -v "<" | xargs curl -s | grep "<title>" works fine for me. (Yeah I know the code's extremely hacky). –  Daniel Beck Dec 2 '11 at 18:10
    
Oh my.. I have to de-cypher that sed part in order to understand what you are doing, still it isn't working with jspider. I will start using perl now. –  Achmed Durangi Dec 5 '11 at 9:39
    
@StephanKristyn I extract URLs from hyperlinks, and since it doesn't work well enough, using grep -v I then remove all remaining lines with HTML tag brackets. Just a simple example that shows the approach works in general. –  Daniel Beck Dec 5 '11 at 9:51
add comment

Try this, it's untested but should work.

for a in $(./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}'); do curl $a | grep myString; done
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.