Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am using a Windows 8 machine and starting with an html page I would like to automatically retrieve the text content from every link on that page that matches a regex. The retrieved content must then be searched for another regex and any matches listed in a table along side the URL from which the match was found.

Currently I am just searching through each URL individually using the Browser search box.

I cannot see any way of doing this with the tools available in any Browsers. I know I could write a script but I figure there may be an application (or a script someone else has written) that does this already.

share|improve this question
It's hard to believe that there is a ready made application that does exactly this. Please rather just ask about what you need to accomplish. Also, it would be good to mention your operating system and what you've already tried. – slhck Sep 25 '13 at 6:23
Has this question violated some policy? Why the down vote? – david004 Sep 25 '13 at 6:27
We expect questions to show some effort into finding a solution, including your attempts (see How to Ask). Also, it's not clear what your OS is, so any answer would be a shot in the dark. – slhck Sep 25 '13 at 6:30

Today I feel as your Mechanical Turk, so I wrote some lines of bash script.

Get all links in a $MAINPAGE:

wget $MAINPAGE -O - | sed 's%<a%\n&%g' | sed 's%.*href=["\']%["\'].*%'

Loop through them and search for the regex:

for LINK in $(wget $MAINPAGE -O - | sed 's%<a%\n&%g' | sed 's%.*href=["\']%["\'].*%'); do
  # abort grepping after first match and return the
  # count (number of matches, which is then 0 or 1),
  # if count > 0 then print the LINK url.
  if [ $(wget $LINK -O - | grep -c -m 1 -e 'I_AM_A_REGEX') -gt 0 ]; then
    echo $LINK

PS: Untested!

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.