Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I want to parse out certain URLs from a saved html-file with a simple bash-script/command, so i can download them via wget laters.
What i have so far is:

awk 'BEGIN{ RS="<a *href *= *\""} NR>2 {sub(/".*/,"");print; }' index.html >> url-list.txt

which writes every link that is inside my html-file into a neat txt-document.

What i need, however, are only specific links

  • from a certain domain
  • all .ogg files for example

everything that in the end would look like:

http://www.foo.com/(randomfolder)/(randombasename).ogg

or

http://subdomain.foo.com/(randomfolder)/(anotherrandomsubfolder)/(randombasename).ogg

thanks in advance!

share|improve this question
    
Need to do this with pure Bash/sed/awk or are scripting languages fine too? –  slhck Apr 17 '12 at 13:56
    
would need it with bash so i can merge it easily with other scripts i wrote, thanks. I'm pretty new to this... –  Kai Apr 17 '12 at 13:59

1 Answer 1

up vote 3 down vote accepted

You say you need to do it "in Bash", but you seem to mean "in a script" and not "using pure Bash syntax" (there is a difference). I assume you want the "in a script" meaning.


If you have saved all the links on separate lines in a document, you can pick out all links on the domain http://www.example.com/ with e.g.

grep "^http://www\.example\.com/" filewithlinks.txt

or all links ending with .ogg with

grep "\.ogg$" filewithlinks.txt

(The backslash preceding periods is to escape the "." sign which otherwise means "any character". "\." instead mean a literal period. Without escaping you'll match links ending with e.g. "logg" as well.)

You could also do line matching directly in the awk command, but that would be more complicated to change from time to time, I believe. The easiest way is to save the complete list of links in a file as you do and then just search the file with for example grep as above. Then you also won't have to download and parse the document again if you want to change matching pattern.

share|improve this answer
    
yes, thanks, i mean, doing it with a simple bash script, using awk, sed, or something else (as implied by the tags). ----- I tried your suggestion on my saved html-page, but it didn't work. My script above parses (reads out) every URL that it finds inside the a href tags of the html file and copies them to a plain txt file, line by line. You might want to try our scripts to see what i mean. For example, save this page, then only keep the links that follow meta.superuser.com or superuser.com/users for a start... –  Kai Apr 17 '12 at 14:37
    
@Kai: I tried it on a saved HTML page. Your awk script rendered a file with one link per line, which is what I called filewithlinks.txt above. Using grep as above on this file as I wrote (nb: not index.html) worked as I described, and should for you as well. Look up some simple grep examples if this tool is new for you (it is very simple: it prints all lines that match the given expression. "^" means beginning of line, "$" means end of line). –  Daniel Andersson Apr 17 '12 at 14:46
    
oic, thanks, I'm sorry, overread that. I have to combine the two then. I tried grep "^http://*superuser\.com/" filewithlinksfromsuperuser.txt >> allsuperuserlinks.txt but it only got me the superuser.com links, when i wanted the subdomains blog.superuser.com and meta.superuser.com too... How do i use the asterisk right? (insisting because it was part of my question, sorry :) Any way to grep links that have a certain amount of numbers in it, like www.foo.com/12345 and www.foo.com/subfolder/123456. Or should i open a new question? –  Kai Apr 17 '12 at 15:09
    
@Kai: This is all in the functionality of grep (in this specific case: use .*, since "." means "any character", and "*" means "the previous character zero or more times"). You should read up on regular expressions for more advanced cases, but the sky is the limit (even more so when using grep -E), which makes it hold too much information to cram into a comment field :-) . –  Daniel Andersson Apr 17 '12 at 15:24
    
okay, thanks for this solution. --- the grep i am looking for would have been grep "^http://.*superuser\.com/" filewithlinksfromsuperuser.txt >> su-subdomainlinks.txt and grep "^http://superuser\.com/\users/[0-9]\{3,6\}" filewithlinksfromsuperuser.txt >> all su-users-links.txt –  Kai Apr 17 '12 at 15:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.