The tag has no wiki summary.

learn more… | top users | synonyms (3)

-1
votes
1answer
28 views

List all links of one website on other website [closed]

There are two websites. 1) www.aa.com 2) www.bb.com Now there are so many link of aa.com and its webpages on bb.com like aa.com/blhahhaa.html aa.com/beautifulday.html aa.com so i want to find ...
0
votes
1answer
12 views

Diff content of a website recursively

I have a lot of tools I'm using and want to be able to monitor their changes through their documentation. An example would be Vagrant Documentation - https://docs.vagrantup.com/v2/. I want a solution ...
1
vote
1answer
61 views

Mirroring a web site having pages that uses simple JavaScript [duplicate]

There are simple and great web site download tools that allow to create a local mirror of simple sites having no JavaScript infrastructure behind. However, sometimes there cases when a site looks ...
0
votes
0answers
27 views

Downloading site with anchor tags HTTrack

When trying to download this site, HTTrack doesn't grab any of the content from within the tabs. How would I get this?
-1
votes
1answer
33 views

Compiling a list of links on a website and their validity [closed]

I would like to crawl my website and generate a list of internal and outbound links, and what their final destination is and HTTP Code (at least for internal links). How can I do that? Please note: I ...
1
vote
1answer
41 views

How would I scrape text from a site? [closed]

Schemer is closing down, and I'm trying to preserve the text because there are many good ideas on the site. I'd like to get the text from within each of the links listed on the sitemap. Is there a ...
1
vote
1answer
462 views

How to crawl using wget to download ONLY HTML files (ignore images, css, js)

Essentially, I want to crawl an entire site with Wget, but I need it to NEVER download other assets (e.g. imagery, CSS, JS, etc.). I only want the HTML files. Google searches are completely useless. ...
1
vote
0answers
16 views

Scan all available requests on website [closed]

I would like to scan some website for all available requests/api requests that can be done on this server, is there any way to do that? Thanks
0
votes
1answer
76 views

Wget getting responce 403

I am using API that has some limit of requests in an hour. But my script does all at one time so I lose about 1/3 of requests because I get 403. Is there any way to check response of wget and if I ...
1
vote
2answers
547 views

Web scraping / crawling a particular Google book

For my work, I need to scrape the text from a large book on Google Books. The book in question is a very old book and is out of copyright. The book is a Gazetteer of the World. We will be putting the ...
0
votes
1answer
315 views

How to crawl your own website to save to cache

I'm using Squid, a caching program, to cache my website. However to do so it appears that each page has to be accessed at least once before Squid can cache it. My question is, is there a program which ...
0
votes
1answer
287 views

recursively downloading all the folders and subfolder from webpage

I would like to download a directory from webpage. i am trying to do using wget and crawl but i am not able to download the whole directory as it was there in web-page or in any-other way. suppose the ...
1
vote
0answers
63 views

Looking for an application/tool for spidering a website that loads more content upon scrolling

I'm looking for an application/tool that will allow me to spider a website where you have to scroll down vertically repeatedly for the whole page to load. I need to be able to define the starting ...
0
votes
1answer
129 views

Windows - Crawl URL and grab links

I have recently signed up to a large video site as a premium member and I want to generate a list of the video files on the site so that I can put them into my download manager and download the ...
0
votes
2answers
270 views

Streaming Video Bulk Download

There is a video series I have been watching: http://english.cntv.cn/program/learnchinese/growingwithchinese/index.shtml I have tried a few plugins and utilities to download the episodes with varying ...
2
votes
2answers
4k views

How can I scrape specific data from a website

I'm trying to scrape data from a website for research. The urls are nicely organized in an example.com/x format, with x as an ascending number and all of the pages are structured in the same way. I ...
0
votes
5answers
4k views

Website crawler/spider to get site map [closed]

I need to retrieve a whole website map, in a format like : http://example.org/ http://example.org/product/ http://example.org/service/ http://example.org/about/ ...
2
votes
1answer
567 views

Is there a graphical web crawler that indexes a site in excel?

I am trying to recreate the product catalogue for an ecommerce site as an excel table. Basically i want to fetch product name, brand name, price, and any associated tags and store it in an excel ...
2
votes
1answer
212 views

wgt downloads all files except for the images i want

i have trouble using wget to download images from a gallery. as start i use the overwiew page. it has thumbs that link to the individual pages with the large images. here is the script i use: wget ...
4
votes
1answer
111 views

Extract data from an online atlas

There is an online atlas that I would like to extract values from. The atlas provides a tool ('Query') to extract values when you click a location or enclose a region on the map, or you can specify ...
1
vote
1answer
483 views

Spider/crawl a website and get each URL and page title in a CSV file

I am moving from an old ASP shopping cart site to a Drupal/Ubercart site. Part of this move is to ensure that old links will redirect to the new ones. To do that all I need is some way to get a list ...
0
votes
1answer
633 views

Scan and map website and log all links that have “particular-string” in them

Is there a tool that would scan a website and log all links that have a particular string in them? I see Teleport Pro, but it seems like it does make a copy of the site, but I only need links.
0
votes
0answers
50 views

Looking to backup content from online class [duplicate]

Possible Duplicate: How can I download an entire website? Basically, I took an online class, and now I want to spider the content, and back it up locally. Any Suggestions on ...
2
votes
3answers
1k views

Is it possible to discover all the files and sub-directories of a URL?

I wonder if there is a software I can use to discover all the files and sub-directories given a URL? For example, given www.some-website.com/some-directory/, I would like to find all the files in ...
2
votes
2answers
134 views

Which sites reject crawler requests?

Is there any site which can be rejected by a crawler? I am using the Burp Suite crawler for the time being to crawl the sites. I want to know when and in which cases a crawler fails to retrieve the ...
0
votes
1answer
32 views

How can I search the Internet for sites containing keywords in HTML (not text)?

I want to search the Internet for sites using a certain widget. Google has already crawled the Internet but their standard search trims any pieces of code and leaves only human readable text to ...
3
votes
0answers
2k views

wget: recursively retrieve urls from specific website

I'm trying to recursively retrieve all possible urls (internal page urls) from a website. Can you please help me out with wget? or is there any better alternative to achieve this? I do not want to ...
1
vote
1answer
1k views

Firefox addon to download a whole site and one step more

Do you know any Firefox addon that could download a whole website and download all the sites from the links on the first website? I mean also all images and so on.
0
votes
2answers
639 views

Can storing 300k files in one folder cause problems?

I'm crawling a large website (over 200k pages) using wget (is there a better tool btw?). Wget is saving all the files to one directory. The partition is HFS (I think), will it cause problems if I ...
7
votes
5answers
24k views

Using Wget to Recursively Crawl a Site and Download Images

How do you instruct wget to recursively crawl a website and only download certain types of images? I tried using this to crawl a site and only download Jpeg images: wget --no-parent --wait=10 ...
1
vote
3answers
191 views

Extracting information from web page in given interval

I have a problem. I need to extract something like 800 records and it would take days to do that manually. The information may be taken by inserting few fields into form (always the same) and then a ...
1
vote
1answer
489 views

Crawling a large directory with wget with two links pointing at the same thing

I'm trying to crawl a directory on a website and basically download everything in it. The structure is simple enough(but there are also multiple folders), but there is one thing that makes wget choke ...
0
votes
0answers
155 views

web spidering/crawling, can i do it or just search engines?

i already had a question answered about web-scraping with wget. but as i read a little more, i realize i may be looking for a web-crawling program. particularly the part about web-crawlers being able ...
6
votes
2answers
6k views

Convert web pages to one file for ebook

I want to download HTMLs (example: http://www.brpreiss.com/books/opus6/) and join it to one HTML or some other format that i can use on ebook reader. Sites with free books don't have standard paging, ...
0
votes
1answer
587 views

Extract text from web

I'm looking for an easy toll which can crawl the web I give it to and extract all text elements from that. It will be nice if it can do: plain text, alt and title for images, header section. All of ...
7
votes
4answers
3k views

How “legal” is site-scraping using cURL? [closed]

Recently I was experimenting with the cURL, and I found lot is possible with it. I built a small script that crawls a musical site, which plays online songs. On the way of my experiment, I found that ...
3
votes
4answers
5k views

What do I use to download all PDFs from a website?

I need to download all the PDF files present on a site. Trouble is, they aren't listed on any one page, so I need something (a program? a framework?) to crawl the site and download the files, or at ...
1
vote
1answer
409 views

Command-line HTTP crawler for Windows? [duplicate]

Possible Duplicate: How can I download an entire website Would somebody have a recommendation for a web site crawler that can be invoked and equipped with settings from the command line? ...
0
votes
1answer
276 views

Google Indexed an Unlinked Page [closed]

Google indexed a page on a site of mine that was not linked from any other page, ever. No one has ever put a link to it, and the directory contents were not browsable. How could this happen? I thought ...
1
vote
1answer
320 views

What is the best way to archive (spider) a site that is going to be removed? [duplicate]

Possible Duplicate: How can I download an entire website Three different blogs that I read have recently announced that they are going to be discontinued and removed from the web. Although ...
0
votes
1answer
270 views

wget: Turn Off Forced .html Retreival

When performing a recursive download, I specify a pattern via the -R parameter for wget to reject, but if this file is a HTML file, wget downloads the file regardless of whether or not it matches the ...
2
votes
2answers
321 views

Tool to recursivly convert a HMTL file to PDF?

Are there any tools which not only convert a HTML file to PDF but also follow links, so that in the end I get 1(!) PDF file which contains all html files?
1
vote
1answer
322 views

Looking for web spider/download program which can use existing browser cookies and can process Javascript

I'm looking for a program to spider a web site* and download the content to disk. I've got the following requirements though which appear to trip up the programs I've tried: The site's logon ...
0
votes
1answer
837 views

I installed and ran Heritrix Web Crawler. It stored data in .arc.gz files

If you have used Heritrix Web Crawler, I'd really appreciate your help. 3 questions: An arc file probably contains source codes of MANY pages in there. How do I figure out which is which? How do I ...