Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

All, I want to extract all URLS mentioned in code statements in all js files in a folder. For e.g. if a js file contains this piece of code:

var myURL="http://yahoo.com/signup"  
//var myURL="http://yahoo.com/sign" --commented out 

then when my script is run, the output should be just "http://yahoo.com/signup". Currently I'm using grep to extract the URLS . The output contains both the statements in the output. I don't want the commented URL in the output.

Currently I'm using a basic grep statement

 cat somefile.js | grep "http[s]\?://"  

All the URLS will be either http/https and will be declared in statements in js files.

I would like to fine tune this grep statement to output only the URLS (excluding the commented URLs) and not the complete statement. How can I fine tune this grep statement?

share|improve this question
add comment

2 Answers

How large is your file / how many do you have?

A very simple solution I use for this type of thing is excel. You could just copy / paste the file (as long as there's only a couple) into excel, then sort the column. You should find all your vars in one place. Copy/paste those into a new sheet, and split the column on '"'. This should give you a column of your urls.

Of course, this isn't practical for many files -- a script would work better there.

EXAMPLE

I find this is particularly useful for parsing json by hand. Here's an example of how I typically use this. Here's a snippit from a json array:

{'english' => 'the animals, 'spanish' => 'los animales', 'image_url' => 'http://www.resourcesgraphics.com/images/Animals-Set-icons.jpg},
{'english' => 'the bear, 'spanish' => ' el oso  , 'image_url' => 'http://images.nationalgeographic.com/wpf/media-live/photos/000/005/cache/grizzly-bear_566_600x450.jpg},
{'english' => 'the beaver, 'spanish' => 'el castor, 'image_url' => 'http://images.nationalgeographic.com/wpf/media-live/photos/000/004/cache/beaver_457_600x450.jpg},
etc... (I have ~2000 entries in this file.)

To get all the URLs out, I just copy/paste the entire file into excel, and use Data>Text to Columns>Delimited. If you split on a quote, it'll pull out the URLs into their own column.

This also goes the other way. (I do this much more often.) If you wanted to make the table above, you could use this process in reverse. To quickly fill columns, I just make a first column like this:

{'*english*' => '*the animals*, '*spanish*' => '*los animales*', '*image_url*' => '*http://www.resourcesgraphics.com/images/Animals-Set-icons.jpg*},

(* = column division)

So, now we have column1 = "{'", column2 = "english", column3 = "' => '", etc... columns 1, 3, 5, etc are fillers, and the data goes in the middle. I'd then fill in all the data, and copy in the fillers. A quick way to fill in filler data is to double click on the bottom right handle of the top filler cell. This should duplicate the data down the column to the end of your data.

After you've made your spreadsheet, save it. You can use it again later. When you need your data, just copy all, and paste into text editor.

There's usualy a bunch of pesky tabs left over from the columns, but those can be removed with a simple find/replace.

Hope that helps -- let me know if you'd like any more info! - L

share|improve this answer
    
Can you elaborate? Maybe a small example/screenshot. That is an interesting approach! –  iglvzx Mar 13 '12 at 18:46
    
Surely! I've updated the answer -- let me know if it's clear / helpful. –  Loren Rogers Mar 13 '12 at 20:42
    
@lrog: I have to loop through 10-15 JS files in a folder with each file having around 400-500 Lines max. –  smokinguns Mar 13 '12 at 21:02
    
@smokinguns Yah, for something like that I figure it may be fastest just to use excel. If you had like > 20 files, I'd say put together a script. For my answer, I assumed that all the urls were defined in these documents, and imported. Are these complicated source-code files? Please describe your data. –  Loren Rogers Mar 14 '12 at 3:39
    
@smokinguns Actually, if you're just having trouble sorting out the real urls from the commented ones, couldn't you just put them in excel and sort them? Then you could split on columns and get your data out. –  Loren Rogers Mar 14 '12 at 3:41
add comment

Here is a PERL solution:

perl -ne '/^[^\/\/]*"(http.+?)"/ && do{print "$1\n"}' somefile.js 

EXPLANATION:

  • perl -ne : Loop through the input file line by line (-n) and run the script given on the command line (-e)

  • /^[^\/\/]*"(http.+?)"/ : Look for lines that do not have // before the first http and save the characters following the http up to the first ". The pattern in parentheses () will be saved as $1.

  • && do{print "$1\n"} : If the lines matches the regex, print the captured pattern (the URL).

CAVEATS:

This is a very simple solution that assumes you have only one URL per line and that all URLs are quoted. Please let me know if you need a more sophisticated version.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.