Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a "source.txt" file which contains list of some URLs. For example:


I want to retrieve each link inside "source.txt" and search through the html of each and extract all links from them which contain "/gp/product" and then store them in "extracted.txt" file, which would be similar to:


I am using Windows 7 (64 bit) and Cygwin, so I can run Linux commands as well.

share|improve this question
You might want to rephrase your question. The first time I read it, I thought a simple grep (as in ssmy’s answer) was what you wanted. Now I’ve read your question 2½ times, and I guess you mean that you want to retrieve each of the web pages whose URLs you have, and then search through the HTML for “/gp/product”. Is that what you mean? If so, I believe you should look at wget. – Scott Apr 2 '13 at 3:53
I modified the question. Yes, I mean retrieve and search through the URLs in "source.txt". – Si14 Apr 2 '13 at 4:21
You can use "wget -qO- -i source.txt | grep /gp/product" but that will output the lines containing "/gp/product" with all html-tags etc. – FSMaxB Apr 2 '13 at 12:13
@FSMaxB Thank you. I tried and you are right. The output is the lines with "href="/gp/product/", while it should be "";. Any suggestions how to modify this? – Si14 Apr 2 '13 at 14:11
@Si14 Maybe you can use awk or sed to extract the actual links from this list, but I don't know how to do it. At least it's a first step. – FSMaxB Apr 2 '13 at 15:44

In a bash shell you can use grep. grep "/gp/product/" source.txt >extracted.txt

share|improve this answer
The above command searches for that keyword only in the source.txt and does not open and search the URLs in it. I tried it in Cygwin. I am not sure how to test it on bash shell? Do you have suggestions? – Si14 Apr 2 '13 at 3:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .