I am looking for links (not urls of pages) written in job postings on a particular website. I would like to scan the website and copy all strings beginning with http or www on ALL the pages (about 1000)

I am on windows 7. i dont know how to run scripts. Can anybody suggest a efficient way of doing this?

Would i have to first download all html pages? If so, then what software should i use for downloading and scanning and copying for the string?

  • You need to provvide an example url of a page. For example, they could be http://example.com/list.php?page=n, where n is the page number. Another possibility is http://example.com/pagen.html, again where n is the page number. It could also be that there's a central list of all pages, but the individual page URLs don't have numbers. In that case, a script could grab all URLs to test from the main page, or you may have to manually cpoy every URL in. This kind of information is necessary.
    – Bob
    Mar 17 '12 at 11:24
  • the site is: www.sarkari-naukri.in; now i need to copy all links mentioned in posts on all the pages.
    – M Singh
    Mar 17 '12 at 12:10

When we keep in mind that running scripts is not an option for you, you could take the approach to download the source code of a page (right click -> download source code). You then can open it with e.g. notepad and search the content by pressing [Ctrl] + [F].

Another way would be using the URLStringGrabber: http://www.nirsoft.net/utils/url_string_grabber.html

  • thx for the reply valentin, but i have to scan the whole website and it contains about 1000 pages. so the source code option is out. now the urlstringgrabber also scans only the opened page. opening 1000 pages one by one will take a loooong time. any other options?
    – M Singh
    Mar 17 '12 at 10:17
  • I'd search the web for a decent tool. Use the search term "scan a website for links" in google if you like, this should bring you the desired results. Mar 17 '12 at 10:31
  • Create website map xml-sitemaps.com and parse output xml file Mar 19 '12 at 15:45

You can easly achieve that in Opera, just open left pane - Links, and you can copy all of them to clipboard

  • hi pinger, is that for the page that is currently open? I need to scan the whole website with about 1000 pages.
    – M Singh
    Mar 17 '12 at 10:18

I find the program WinHTTrack to be useful for this purpose. There is a combination of options that allow you to download a single page, but change the URLs to a specific, absolute format so that you can later search the raw HTML and be guaranteed almost all of the links.

  1. After setting the mirror name and progressing to the next screen, change the Action to "Download web site(s)".
  2. Put the URL of the page that contains more web pages in the "Web Addresses: (URL)" box.
  3. Select Options -> Experts Only
  4. Change the "Rewrite Links: internal / external" to "Absolute URI / Absolute URL" (or, if you're only using the page for scraping URLs, "Absolute URL / Absolute URL").
  5. Press OK, then Next, then navigate through the options as usual.

More information about HTTrack can be found on the tag.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.