Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to scrape a website for all sites and append an Paramater, e.g.

www.theSite.com/allpages?myPara=us

Now what I got is wget --recursive to get all sites, but that doesn't append the parameter and the content is useless. I need to do this in a 2nd step:

curl http://theSite.com/allpages?myPara=us | grep -c MyString

So in essence I am looking for a particular String in a entire Website to which I got only access via HTTP.

share|improve this question

1 Answer 1

up vote 1 down vote accepted

I managed now by using brackets and curl like this:

curl http://www.mySite.com/category1/category2/{+cat1,+cat2,+cat3,+cat4,+cat4}?myPara=us | grep -c myString

Of course this won't get me all sites, but one could work out a script with wget, regex, collections and loops.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.