The web-crawler tag has no wiki summary.
0
votes
0answers
37 views
Most efficient way to have a 24/7 link scraper/crawler? [closed]
I am new at this site, and am wondering about the following:
I am building a search engine from scratch and want to have a 24/7 spider.
I will only scrape links from different pages, then visit the ...
0
votes
0answers
8 views
Website crawler to return urls of assets [duplicate]
Does anyone know of a website crawler that will return a list of assets used throughout a site?
I'm migrating some files (on a sub domain) to a new server, but I've lost track of which files are ...
1
vote
0answers
49 views
Looking for an application/tool for spidering a website that loads more content upon scrolling
I'm looking for an application/tool that will allow me to spider a website where you have to scroll down vertically repeatedly for the whole page to load. I need to be able to define the starting ...
0
votes
0answers
60 views
Need tool/program to crawl small subset of facebook (just friends of friends) [closed]
I'm looking for a tool/program that will easily allow me to write a script to crawl facebook, and do the following: Load person A's friends list. Goto each of those friends pages, and see if another ...
0
votes
0answers
101 views
Simple Image Crawler [closed]
I am working on a web application where I have to implement a basic crawler which will fetch images from some 1 Million websites (stored in database).
I have created the script and currently I am ...
0
votes
1answer
66 views
Windows - Crawl URL and grab links
I have recently signed up to a large video site as a premium member and I want to generate a list of the video files on the site so that I can put them into my download manager and download the ...
0
votes
2answers
131 views
Streaming Video Bulk Download
There is a video series I have been watching: http://english.cntv.cn/program/learnchinese/growingwithchinese/index.shtml
I have tried a few plugins and utilities to download the episodes with varying ...
2
votes
2answers
434 views
How can I scrape specific data from a website
I'm trying to scrape data from a website for research.
The urls are nicely organized in an example.com/x format, with x as an ascending number and all of the pages are structured in the same way. I ...
0
votes
5answers
1k views
Website crawler/spider to get site map [closed]
I need to retrieve a whole website map, in a format like :
http://example.org/
http://example.org/product/
http://example.org/service/
http://example.org/about/
...
2
votes
1answer
262 views
Is there a graphical web crawler that indexes a site in excel?
I am trying to recreate the product catalogue for an ecommerce site as an excel table.
Basically i want to fetch product name, brand name, price, and any associated tags and store it in an excel ...
2
votes
1answer
131 views
wgt downloads all files except for the images i want
i have trouble using wget to download images from a gallery.
as start i use the overwiew page. it has thumbs that link to the individual pages with the large images.
here is the script i use:
wget ...
4
votes
1answer
82 views
Extract data from an online atlas
There is an online atlas that I would like to extract values from. The atlas provides a tool ('Query') to extract values when you click a location or enclose a region on the map, or you can specify ...
1
vote
1answer
199 views
Spider/crawl a website and get each URL and page title in a CSV file
I am moving from an old ASP shopping cart site to a Drupal/Ubercart site. Part of this move is to ensure that old links will redirect to the new ones. To do that all I need is some way to get a list ...
0
votes
1answer
416 views
Scan and map website and log all links that have “particular-string” in them
Is there a tool that would scan a website and log all links that have a particular string in them?
I see Teleport Pro, but it seems like it does make a copy of the site, but I only need links.
0
votes
0answers
50 views
Looking to backup content from online class [duplicate]
Possible Duplicate:
How can I download an entire website?
Basically, I took an online class, and now I want to spider the content, and back it up locally. Any Suggestions on ...
1
vote
3answers
580 views
Is it possible to discover all the files and sub-directories of a URL?
I wonder if there is a software I can use to discover all the files and sub-directories given a URL?
For example, given www.some-website.com/some-directory/, I would like to find all the files in ...
2
votes
2answers
88 views
Which sites reject crawler requests?
Is there any site which can be rejected by a crawler? I am using the Burp Suite crawler for the time being to crawl the sites.
I want to know when and in which cases a crawler fails to retrieve the ...
0
votes
1answer
31 views
How can I search the Internet for sites containing keywords in HTML (not text)?
I want to search the Internet for sites using a certain widget.
Google has already crawled the Internet but their standard search trims any pieces of code and leaves only human readable text to ...
3
votes
0answers
633 views
wget: recursively retrieve urls from specific website
I'm trying to recursively retrieve all possible urls (internal page urls) from a website.
Can you please help me out with wget? or is there any better alternative to achieve this?
I do not want to ...
1
vote
1answer
174 views
Firefox addon to download a whole site and one step more
Do you know any Firefox addon that could download a whole website and download all the sites from the links on the first website? I mean also all images and so on.
0
votes
2answers
362 views
Can storing 300k files in one folder cause problems?
I'm crawling a large website (over 200k pages) using wget (is there a better tool btw?). Wget is saving all the files to one directory.
The partition is HFS (I think), will it cause problems if I ...
7
votes
5answers
15k views
Using Wget to Recursively Crawl a Site and Download Images
How do you instruct wget to recursively crawl a website and only download certain types of images?
I tried using this to crawl a site and only download Jpeg images:
wget --no-parent --wait=10 ...
1
vote
3answers
180 views
Extracting information from web page in given interval
I have a problem. I need to extract something like 800 records and it would take days to do that manually.
The information may be taken by inserting few fields into form (always the same) and then a ...
1
vote
1answer
390 views
Crawling a large directory with wget with two links pointing at the same thing
I'm trying to crawl a directory on a website and basically download everything in it. The structure is simple enough(but there are also multiple folders), but there is one thing that makes wget choke ...
0
votes
0answers
133 views
web spidering/crawling, can i do it or just search engines?
i already had a question answered about web-scraping with wget. but as i read a little more, i realize i may be looking for a web-crawling program. particularly the part about web-crawlers being able ...
3
votes
2answers
3k views
Convert web pages to one file for ebook
I want to download HTMLs (example: http://www.brpreiss.com/books/opus6/) and join it to one HTML or some other format that i can use on ebook reader. Sites with free books don't have standard paging, ...
0
votes
1answer
399 views
Extract text from web
I'm looking for an easy toll which can crawl the web I give it to and extract all text elements from that. It will be nice if it can do: plain text, alt and title for images, header section. All of ...
7
votes
4answers
2k views
How “legal” is site-scraping using cURL? [closed]
Recently I was experimenting with the cURL, and I found lot is possible with it. I built a small script that crawls a musical site, which plays online songs. On the way of my experiment, I found that ...
3
votes
4answers
2k views
What do I use to download all PDFs from a website?
I need to download all the PDF files present on a site. Trouble is, they aren't listed on any one page, so I need something (a program? a framework?) to crawl the site and download the files, or at ...
1
vote
1answer
301 views
Command-line HTTP crawler for Windows? [duplicate]
Possible Duplicate:
How can I download an entire website
Would somebody have a recommendation for a web site crawler that can be invoked and equipped with settings from the command line?
...
0
votes
1answer
173 views
Google Indexed an Unlinked Page [closed]
Google indexed a page on a site of mine that was not linked from any other page, ever. No one has ever put a link to it, and the directory contents were not browsable. How could this happen? I thought ...
1
vote
1answer
224 views
What is the best way to archive (spider) a site that is going to be removed? [duplicate]
Possible Duplicate:
How can I download an entire website
Three different blogs that I read have recently announced that they are going to be discontinued and removed from the web. Although ...
0
votes
1answer
238 views
wget: Turn Off Forced .html Retreival
When performing a recursive download, I specify a pattern via the -R parameter for wget to reject, but if this file is a HTML file, wget downloads the file regardless of whether or not it matches the ...
2
votes
2answers
175 views
Tool to recursivly convert a HMTL file to PDF?
Are there any tools which not only convert a HTML file to PDF but also follow links, so that in the end I get 1(!) PDF file which contains all html files?
0
votes
1answer
80 views
Spider a website with variations in the POST parameters [closed]
I need an online or offline app that can take a URL, vary the input value for one parameter according to rules set by the user, and then download each varied url it gets.
For example, if I give it ...
1
vote
1answer
253 views
Looking for web spider/download program which can use existing browser cookies and can process Javascript
I'm looking for a program to spider a web site* and download the content to disk. I've got the following requirements though which appear to trip up the programs I've tried:
The site's logon ...
0
votes
2answers
664 views
I installed and ran Heritrix Web Crawler. It stored data in .arc.gz files
If you have used Heritrix Web Crawler, I'd really appreciate your help.
3 questions:
An arc file probably contains source codes of MANY pages in there. How do I figure out which is which?
How do I ...
