2

I would like to extract links from a numerical sequence of pages like this:

http://example.com/page001.html
http://example.com/page002.html
http://example.com/page003.html
...
http://example.com/page329.html

What I want at the output is a text file with URLs gathered from the links on these pages:

http://www.test.com/index.html
http://www.google.com
http://www.superuser.com/questions

To be clear, I don't want to download the pages, I just want a list of links.

Windows software would be idea, but Linux would be okay too. All I can think of is writing a long batch script with Xidel, but it wouldn't be very robust when encountering errors. Curl can download the range of pages, but then I need to parse them somehow.


Thanks to Enigman for putting me on the right track. I created a Perl script that reads URLs from a file and spits out links matching a string stored in $site:

use warnings;
use LWP;
$site = "twitter.com";

my $browser = LWP::UserAgent->new;
my @ns_headers = (
    'User-Agent' => 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36',
    'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language' => 'en-GB,en;q=0.8',
);

open (URLLIST, 'urls.txt');
while (<URLLIST>) {
    chomp;
    print "# $_\n";
    my $response = $browser->get($_, @ns_headers);
    die "Can't get $_ -- ", $response->status_line
        unless $response->is_success;

    my @urls = $response->content =~ /\shref="?([^\s>"]+)/gi ;
    foreach $url(@urls) {
        if ($url =~ /$site/) {
            print("$url\n");
        }
    }
}
close(URLLIST);

To generate the URL list I made a little batch script:

@echo off
for /l %%i in (0, 15, 75) do @echo http://www.example.com/page_%%i.html

The Perl script just stops on an error, which I prefer. It would be trivial to modify it to just carry on. The User agent and accept data is ripped from Chrome, because some sites don't like anything which looks like a bot. If you are intending to scan sites you do not own please respect the robots.txt and set up a custom user agent.

5
  • I don't understand what you mean. Could you explain it in more detail? Which urls do you want to parse? Do you want simply extract the href property of <a> elements? Where do you get the numerical sequence?
    – Birei
    Aug 16, 2014 at 20:08
  • Say the pages have links to files stored on randomfilehost.com. I want to parse the range of pages and extract all those links. Just the URLs, no need for any of the HTML. Thanks.
    – user237698
    Aug 17, 2014 at 9:13
  • So you mean you magically know a list of URLs built around a one-up number, and you do want to download the pages, but then you want to parse them, extract the href properties of the <a> elements, save those, and discard the rest of the HTML. Right? Aug 19, 2014 at 0:08
  • I just want the links, which does involve downloading the pages but I don't need to store them if that's what you mean.
    – user237698
    Aug 19, 2014 at 8:17
  • how deep do you need to go? Just whatever's on the main page?
    – Journeyman Geek
    Aug 19, 2014 at 9:09

3 Answers 3

3
+50

If you wanted to use code to do this you can do it in Perl using LWP::Simple or Mechanize modules.

The following might have what you are after Find All Links from a web page using LWP::Simple module

This is assuming you are comfortable with using a command line solution using Perl. This works the same on both Windows and Linux platforms. It wouldn't take much to modify to take URL's as parameters from the command line to parse.

1
  • Thanks, this looks like something I can bodge into a workable solution.
    – user237698
    Aug 19, 2014 at 8:25
3

Yep, it's a good ol' bash script. This uses the lynx browser to extract the URLs from the pages and dump them to a text file:

#!/bin/bash
#
# Usage:
#
#   linkextract <start> <end> <pad> <url>
#
#   <start> is the first number in the filename range. Must be an integer
#   <stop> is the last number in the filename range. Must be an integer
#   <pad> is the number of digits the number in the filename is zero-padded to. 
#   <url> is the URL. Insert "<num>" where you want the number to appear. You'll
#         need to enclose the entire argument in quotes

for (( i=${1} ; i<=${2} ; i++ )); do {
    num=$(printf "%04d" ${i})
    url=$(echo ${4} | sed "s/<num>/${num}/")
    lynx -dump -listonly "${url}" | sed -r -n "/^ +[0-9]/s/^ +[0-9]+\. //p"
}; done

You'll need to install the lynx browser, which is available on Debian as the package 'lynx'. The script prints the extracted URLs to stdout. So for the example in your question you would do (assuming you save the script to a file called linkextract):

$ linkextract 1 329 3 "http://example.com/page<num>.html"
1

You can use the Site Visualizer crawler for this work. Download and install it, then click New Project, type your website's URL, click OK, then Start Crawl toolbutton.

After the crawling is completed, double-click All Links report of the Reports tab. You'll be given all links that are present on the website, as well as other info: source/target link URLs, content type (HTML, image, pdf, css, etc.), response, and so on. Select all the table (context menu, or Ctrl+A shortcut), then click Copy Rows with Headers context menu item. After that you can paste the data into an Excel sheet, or a simple text document:

extract all website links

The program has 30-day trial period, but it's full-featured, so you can use it for free for 1 month.

2
  • Thanks. The only issue I see is that it doesn't seem to support the range of pages I need. There are a lot of crawling apps available but they just cover the entire site, which has tens of thousands of pages. I just want to do a range of a few hundred and nothing else.
    – user237698
    Aug 19, 2014 at 8:19
  • You can set such range of URLs by using the Include URLs crawling option.
    – Oleg
    Aug 19, 2014 at 8:22

You must log in to answer this question.