Extract URL from text with multiple URLs using sed

Question

I have a string with a text and several URLs. How can I extract one specific URL (of a spoecific domain) with sed? For example, I have this:

Text foo bar Text foo bar <br /><br /> http://www.this.file <br />http://another.file <br />http://mine.com/this.html <br />http://myURL.net/files/IWANTthis <br />http://www.google.com/thisnot

and sed shall return this: http://myURL.net/files/IWANTthis

Answer 1

There could be some trouble with sed on special cases. As has been suggested in many places (e.g.) - not to use regexps but a html parser engine. One such easily available parser is contained in the text only browser lynx (available on any linux). Then you just extract the urls you want with grep.

lynx -dump -listonly myhtmlfile.html | grep IWANTthis | sort -u

However this will not work on mangled html files (cannot be parsed properly) or text snippets with links. Another simple way is to chain. If you have a text snippet like yours in a text file called st3.txt you can do as follows:

grep http ./st3.txt | sed 's/http/\nhttp/g' | grep ^http | sed 's/\(^http[^ <]*\)\(.*\)/\1/g' | grep IWANTthis | sort -u

Explanation:

grep http ./st3.txt      => will catch lines with http from text file
sed 's/http/\nhttp/g'    => will insert newline before each http
grep ^http               => will take only lines starting with http
sed 's/\(^http[^ <]*\)\(.*\)/\1/g'   
                         => will preserve string from ^http until first space or <
grep IWANTthis           => will take only urls containing your text of interest
sort -u                  => will sort and remove duplicates from your list

Answer 2

You can use grep with an extended regular expression (regex) for this:

grep -Eo '([-+.[:alnum:]]+://)?([-[:alnum:]]+.)*myURL.net(:[[:digit:]]+)?(/[[:graph:]]*)?' <<< '<input_string>'

Explanation of each part of the command and the regex:

• grep -Eo: We call grep with two options.
  • -E: Enable POSIX extended regexes.
  • -o: Print only the matched parts of each line (without this option, grep defaults to printing entire lines that contain matching parts).
• '([-+.[:alnum:]]+://)?([-[:alnum:]]+.)*myURL.net(:[[:digit:]]+)?(/[[:graph:]]*)?': This regex is probably a bit fancier than what you need, so you should simplify or remove parts that are needlessly complex for your purposes. (The complexity is the result of an attempt to give a general solution.)
  • ([-+.[:alnum:]]+://)?: This matches the scheme of the URL. The ? at the end specifies that this matches either once or not at all (so that myURL.net is matched even if no scheme is prepended). We include the slightly unusual characters -+. because RFC 3986 specifies that "[s]cheme names consist of a sequence of characters beginning with a letter and followed by any combination of letters, digits, plus ('+'), period ('.'), or hyphen ('-')" (emphasis mine). If you are sure that http:// will always appear in front of myURL.net, then you can replace this part of the regex with the simple http://.
  • ([-[:alnum:]]+.)*: This matches subdomains. The * at the end specifies that this matches zero or more times (so that subdomains like a.b. in a.b.myURL.net are matched). We include the hyphen (-) because RFC 1035 specifies that "[subdomain labels] must start with a letter, end with a letter or digit, and have as interior characters only letters, digits, and hyphen" (emphasis mine). If you are sure that you don't need to match subdomains, then you can remove this part of the regex.
  • myURL.net: This is a straightforward match for the literal string myURL.net.
  • (:[[:digit:]]+)?: This matches any port designation if it is included in the URL.
  • (/[[:graph:]]*)?: This matches the rest of the URL. [:graph:] matches any visible character.
• <<< '<input_string>': We feed grep (which accepts a file) our input string using a here string. (echo '<input_string>' | is the common alternative.)

Example:

$ grep -Eo '([-+.[:alnum:]]+://)?([-[:alnum:]]+.)*myURL.net(:[[:digit:]]+)?(/[[:graph:]]*)?' <<< 'Text foo bar Text foo bar <br /><br /> http://www.this.file <br />http://another.file <br />http://mine.com/this.html <br />http://myURL.net/files/IWANTthis <br />http://www.google.com/thisnot'
http://myURL.net/files/IWANTthis

Again, the regex I gave above is probably overly complicated for what you need it to do. You should modify it to suit your purposes. Something as simple as the following could work:

grep -Eo 'http://myURL.net(/[[:graph:]]*)?' <<< '<input_string>'

Answer 3

You could use: sed 's/<br\ *\/>/\n/g' html_file | grep myURL.net

Output: http://myURL.net/files/IWANTthis

Basically I'm replacing all the <br /> with a newline character and grabbing the relevant line with grep.

This does not account for all possible (html allowed) variations of the <br> tag, but it handles the ones in your example.

Answer 4

You can parse the file for all urls with sed and then grep for your match.

sed "s/http/\nhttp/g" your.html | sed -n "s#\(.*\)\(http.*//[a-Z0-9./-]*[^a-Z/]\)\(.*\)#\2#p;" | grep IWANTthis

In the first pass sed replaces all the http with a prepended newline to make things easier for sed on the second pass where sed rips the urls. This regex generally works for me but you may need to adjust it to suit your needs.

Note: You could do this in one pass but it will get complicated with cryptic potentially hard to maintain seldom used sed commands.