0

I have a 3000 line source code text file of a page. How I can extract specific urls similar to down below and delete rest source code ?

https://d17nygptf7ayu.cloudfront.net/photos/0b1983ccf6bb5674f93b.jpg

As I went thru code, there are 60 links similar to above link with only change in *.jpg names.

as a beginner I have least idea where to start and which tool I should use on linux command line. Also how i can do this in notepad++?

Regex I come up with https\://d17nygptf7ayu\.cloudfront\.net/photos/(?:(?!\.jpg)(?:.|\n))*\.jpg

2
  • If the name is always 20 hex characters: https\://d17nygptf7ayu\.cloudfront\.net/photos/[0-9a-f]{20}.jpg, or if you want to allow anything before the .jpg: https\://d17nygptf7ayu\.cloudfront\.net/photos/[^\.]+.jpg
    – bitinerant
    Mar 29, 2021 at 22:07
  • Yes, all images have 20 hex characters. But is there any tool to extract similar urls from a text file? I can do this in notepad++. But as beginner, I am looking for command line tool to do this.
    – Roxion
    Mar 30, 2021 at 4:00

1 Answer 1

1

This regular expression will match the URLs you want:

https\://d17nygptf7ayu\.cloudfront\.net/photos/[0-9a-f]{20}.jpg

The [0-9a-f] above matches any lower-case hex digit, and {20} matches exactly 20 of them.

The classic and oh-so-useful Linux tool that searches based on regular expressions is grep. One usually uses the -E flag, which causes it to use extended regular expressions. The -o flag causes it to output only the matching string. (Normally grep would output the entire line which contains the matching string.) For a full description of how to use grep, type man grep.

So to generate a list of all the matching URLs:

grep -Eo 'https\://d17nygptf7ayu\.cloudfront\.net/photos/[0-9a-f]{20}.jpg' long_text_file.html >output_list_of_urls.txt

Note that, if the file you list in place of output_list_of_urls.txt already exists, it will be overwritten. Also note that the regular expression is in quotes. If your input or output filenames contain spaces or other special characters, they need quotes too.

3
  • This is faster than doing it on notepad++. As I have more than 300 html files from where I wanted to extract. for f in *.txt;do grep -E -o 'https\://d1a0n9gptf7ayu\.cloudfront\.net/photos/[0-9a-f]{32}.jpg' "$f" > "$(basename "$f" .txt)";done took just 3 seconds.
    – Roxion
    Mar 30, 2021 at 6:41
  • One thing on notice that links are not in order same as in html file.
    – Roxion
    Mar 30, 2021 at 6:50
  • The links should be in the same order as in the input file; grep process the file from start to finish. Nice for loop, above. As an alternative, if you wanted the output combined in a single file, you could use >>output_list_of_urls.txt rather than > "$(basename "$f".txt and the output will be appended to that file.
    – bitinerant
    Mar 30, 2021 at 18:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .