I have a string with a text and several URLs. How can I extract one specific URL (of a spoecific domain) with sed? For example, I have this:

Text foo bar Text foo bar <br /><br /> http://www.this.file <br />http://another.file <br />http://mine.com/this.html <br />http://myURL.net/files/IWANTthis <br />http://www.google.com/thisnot

and sed shall return this: http://myURL.net/files/IWANTthis

  • Don't you do this with grep?
    – 174140
    Dec 10, 2013 at 15:10
  • If you know how, yes please!
    – tzippy
    Dec 10, 2013 at 15:11
  • It's easy dude, do some trial/error work starting with something like | sed s/<br \/>/<br \/>\n/g | grep "myURL\.net"
    – 174140
    Dec 10, 2013 at 15:25
  • The exact works depend on your complete mileage. If you are moving to GNU/Linux, start by reading Eric S. Raymond's The Art of Unix Programming.
    – 174140
    Dec 10, 2013 at 15:28

4 Answers 4


There could be some trouble with sed on special cases. As has been suggested in many places (e.g.) - not to use regexps but a html parser engine. One such easily available parser is contained in the text only browser lynx (available on any linux). Then you just extract the urls you want with grep.

lynx -dump -listonly myhtmlfile.html | grep IWANTthis | sort -u

However this will not work on mangled html files (cannot be parsed properly) or text snippets with links. Another simple way is to chain. If you have a text snippet like yours in a text file called st3.txt you can do as follows:

grep http ./st3.txt | sed 's/http/\nhttp/g' | grep ^http | sed 's/\(^http[^ <]*\)\(.*\)/\1/g' | grep IWANTthis | sort -u


grep http ./st3.txt      => will catch lines with http from text file
sed 's/http/\nhttp/g'    => will insert newline before each http
grep ^http               => will take only lines starting with http
sed 's/\(^http[^ <]*\)\(.*\)/\1/g'   
                         => will preserve string from ^http until first space or <
grep IWANTthis           => will take only urls containing your text of interest
sort -u                  => will sort and remove duplicates from your list
  • Nice! You explained very well.
    – xyz
    Sep 14, 2014 at 17:14
  • 1
    I use lynx as well, I find it useful but I've found sed actually rips more urls by itself since sed doesn't care about formatting and keeping up with the HTML standards.
    – Adam D.
    Nov 15, 2015 at 23:38
  • Yes, sed just processes text, does not parse at all. For this reason - to minimise (but I cannot exclude them totally) such problems I first extract lines with http in them, make more lines by chopping before each http then match those new lines until first space or angle bracket - since hyperlink cannot contain those and then hopefully I only have list of links, from which I extract those with text of interest. Line breaks inside URLS would break this of course.
    – r0berts
    Feb 3, 2019 at 20:05

You can use grep with an extended regular expression (regex) for this:

grep -Eo '([-+.[:alnum:]]+://)?([-[:alnum:]]+.)*myURL.net(:[[:digit:]]+)?(/[[:graph:]]*)?' <<< '<input_string>'

Explanation of each part of the command and the regex:

  • grep -Eo: We call grep with two options.
    • -E: Enable POSIX extended regexes.
    • -o: Print only the matched parts of each line (without this option, grep defaults to printing entire lines that contain matching parts).
  • '([-+.[:alnum:]]+://)?([-[:alnum:]]+.)*myURL.net(:[[:digit:]]+)?(/[[:graph:]]*)?': This regex is probably a bit fancier than what you need, so you should simplify or remove parts that are needlessly complex for your purposes. (The complexity is the result of an attempt to give a general solution.)
    • ([-+.[:alnum:]]+://)?: This matches the scheme of the URL. The ? at the end specifies that this matches either once or not at all (so that myURL.net is matched even if no scheme is prepended). We include the slightly unusual characters -+. because RFC 3986 specifies that "[s]cheme names consist of a sequence of characters beginning with a letter and followed by any combination of letters, digits, plus ('+'), period ('.'), or hyphen ('-')" (emphasis mine). If you are sure that http:// will always appear in front of myURL.net, then you can replace this part of the regex with the simple http://.
    • ([-[:alnum:]]+.)*: This matches subdomains. The * at the end specifies that this matches zero or more times (so that subdomains like a.b. in a.b.myURL.net are matched). We include the hyphen (-) because RFC 1035 specifies that "[subdomain labels] must start with a letter, end with a letter or digit, and have as interior characters only letters, digits, and hyphen" (emphasis mine). If you are sure that you don't need to match subdomains, then you can remove this part of the regex.
    • myURL.net: This is a straightforward match for the literal string myURL.net.
    • (:[[:digit:]]+)?: This matches any port designation if it is included in the URL.
    • (/[[:graph:]]*)?: This matches the rest of the URL. [:graph:] matches any visible character.
  • <<< '<input_string>': We feed grep (which accepts a file) our input string using a here string. (echo '<input_string>' | is the common alternative.)


$ grep -Eo '([-+.[:alnum:]]+://)?([-[:alnum:]]+.)*myURL.net(:[[:digit:]]+)?(/[[:graph:]]*)?' <<< 'Text foo bar Text foo bar <br /><br /> http://www.this.file <br />http://another.file <br />http://mine.com/this.html <br />http://myURL.net/files/IWANTthis <br />http://www.google.com/thisnot'

Again, the regex I gave above is probably overly complicated for what you need it to do. You should modify it to suit your purposes. Something as simple as the following could work:

grep -Eo 'http://myURL.net(/[[:graph:]]*)?' <<< '<input_string>'
  • It's really a bad idea to use regexp against XML (or HTML if you wish). See above answer why.
    – 0andriy
    Aug 30, 2021 at 8:39

You could use: sed 's/<br\ *\/>/\n/g' html_file | grep myURL.net

Output: http://myURL.net/files/IWANTthis

Basically I'm replacing all the <br /> with a newline character and grabbing the relevant line with grep.

This does not account for all possible (html allowed) variations of the <br> tag, but it handles the ones in your example.


You can parse the file for all urls with sed and then grep for your match.

sed "s/http/\nhttp/g" your.html | sed -n "s#\(.*\)\(http.*//[a-Z0-9./-]*[^a-Z/]\)\(.*\)#\2#p;" | grep IWANTthis

In the first pass sed replaces all the http with a prepended newline to make things easier for sed on the second pass where sed rips the urls. This regex generally works for me but you may need to adjust it to suit your needs.

Note: You could do this in one pass but it will get complicated with cryptic potentially hard to maintain seldom used sed commands.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .