Questions tagged [web-crawler]
The web-crawler tag has no usage guidance.
76
questions
4
votes
2
answers
1k
views
Why website copy tools like Cyotek WebCopy and HTTrack cannot find files that search engines like Google can?
I would like to keep the target website private, but here are some details:
It's a personal (as in single-author) public documentation / portfolio / blog sort of website
It seems to be hosted using ...
2
votes
1
answer
862
views
How to recover emails from a closed server [closed]
I have an old email account on Altern.org. Unfortunately, the server had been closed and I didn't succeed to contact the moderator to retrieve my emails.
Is there any archive server (as web.archive....
0
votes
0
answers
3k
views
WinHTTrack Website Copier for morroring website with login (password protection)
I have a login protected site to which I know the login username and password. I need to use WinHTTrack to clone that site.
I came across many tutorials on the internet but none of them are working ...
1
vote
1
answer
2k
views
Facing trouble in Configuring Tor to hash a new password in windows 10
I have tried and read various articles over the internet about hashing a new password for Tor.
My tor is installed in local disk D
I used the following commands in the command prompt
D:\softwares\tor\...
0
votes
0
answers
33
views
Index a whole website that has blocked Google?
I tried to do a site:site.com [search terms] on Google but site.com has blocked Google from indexing it via its robots.txt. How can I get around this? Can I download and index the whole site myself ...
1
vote
1
answer
206
views
Web crawling/scraping a list of all files (which are publicly available) instead of going via 5 clocks from home page for each file
Summary of the problem
I am trying to create a list (and potentially download) of all files (100s) from the following address: https://eba.europa.eu/regulation-and-policy
This is something I would use ...
0
votes
0
answers
259
views
Slow requests when web scraping on a linux Virtual Machine
I have to make a bunch of requests to scrape a website.
My computer alone would take too much time to make all theses requests, so I run my Python script on 7 virtual machines instead.
On my Windows ...
0
votes
1
answer
803
views
Apache Nutch indexing error in Solr
Indexing 1000/1000 documents
Deleting 0 documents
Indexing 1000/1000 documents
Deleting 0 documents
Indexing job did not succeed, job status:FAILED, reason: NA
Indexer: java.lang.RuntimeException: ...
0
votes
1
answer
549
views
What exactly does 'black list" mean in wget?
I'm running the following command
wget -d -e robots=off --recursive -l 10 -w 6.8 -t 3 -nc --random-wait -T 10 -R "*.js, *.css, *.jsp, *.mp3, *.mp4, *.swf, *.apk, *.pdf, *.css, *.doc, *.docx, *.xls, *....
1
vote
2
answers
4k
views
No module named 'scrapy.conf'
I'm trying to execute Twitter scraper code from: https://github.com/jonbakerfish/TweetScraper
When I run the command scrapy list in the command prompt to ensure the scraper is properly set up, I get ...
1
vote
0
answers
38
views
How to filter out "Crawlers"/Image Proxies when tracking email content or links and get ONLY real clicks of the user
Currently I am working on a feature with which I should be able to track E-Mails and the links in it. So when a user clicks on a link in an image I make some entries in the database so that I can make ...
1
vote
1
answer
391
views
wget decides not to load because of black list
I'm trying to make a full copy of a web site; e.g.,
http://vfilesarchive.bgmod.com/files/
I'm running
wget -r -level=inf -R "index.html*" --debug http://vfilesarchive.bgmod.com/files/
and getting,...
1
vote
1
answer
154
views
How can we know which URLs can be crawled as robots.txt tells if we don't know to which folder a URL belong to?
I'm going to code a web crawler but before I want to know what is going to be possible to crawl.
Tell me if I'm wrong, but in robots.txt websites indicate folders not URLs that can and can't be ...
1
vote
0
answers
47
views
Any Chrome extension or plugin can automatically save webpages viewed?
I am looking for a software or browser extension (better with Chrome) which can save all the webpages I viewed automatically on local.
As far as I know, there is one with Firefox called ScrapBook, ...
2
votes
2
answers
228
views
Is a website that is not linked anywhere completely hidden?
I want to create a webpage that only me, and another person have access to view.
I don't want to use any authentication on it (username/password, certificates etc).
My question is:
If I serve the ...
1
vote
1
answer
1k
views
How to crawl a large list of urls? [closed]
I have a massive list of urls. It looks something like this:
www.example.com/api/users/1.json
www.example.com/api/users/2.json
www.example.com/api/users/3.json
And continues about 400,000 times. The ...
1
vote
0
answers
105
views
How to download a website recursively which is behind Google auth?
I want to download a website recursively that is behind Google auth. I found lots of links in the Chrome store that allow me to download the currently open webpage, but none of them follow links.
...
0
votes
1
answer
251
views
How to do a batch input from a web server?
I am trying to load a protein structure file (e.g. this) to this website. It only allows us to load one structure at a time. But I have hundreds of structure files to load. Is there a way that I can ...
18
votes
1
answer
153k
views
How to save all files/links from a telegram chat/channel?
I want to save ALL http(s) links and/or files, posted to some telegram chat (private or group) or channel (like mailing list).
I need an analog of TumblOne (for tumblr) VkOpt (able to save chating ...
0
votes
1
answer
157
views
Write URL's to a text file that match a pattern
I'm trying to parse the nickjr.com site, specifically "http://www.nickjr.com/paw-patrol/videos/" URL for full episodes only.
nickjr.com has their site laid out by show title, then by games or videos,...
-2
votes
3
answers
2k
views
How to implement anti-scraping mechanisms for my Amazon S3 based site?
I have a few static web pages hosted on amazon S3 that are very frequently updated. I want to implement few anti-scraping mechanisms like banning IP's which are making too many requests or making ...
-1
votes
1
answer
2k
views
Crawl website for files
Hi I'd like to download all PDF's from http://www.allitebooks.com/ and would like to use wget. my command is "http://www.allitebooks.com/" -P "C:\dummydir" -c -A pdf -r but I believe it cannot follow ...
0
votes
0
answers
87
views
Centos 7 - Apache banning my web application security crawler
I'm trying to run a scan on my website using Vega, but after the crawler has sent a couple of requests my ip address gets blocked for 30 minutes. This is not the fault of fail2ban(which is installed), ...
2
votes
1
answer
3k
views
How to find the pages that links to a specific page?
I have the following page
http://www.fda.gov/downloads/scienceresearch/fieldscience/laboratorymanual/ucm092156.pdf
I would like to find the pages on www.fda.gov that links to this page? How can I do ...
4
votes
1
answer
542
views
Finding pages on a webpage that contain a certain link
Google does a good jobs finding relevant information.
Say I google: FDA's opinion on ISO-9001
Then it finds a link to a PDF on fda.gov
http://www.fda.gov/downloads/MedicalDevices/...
1
vote
0
answers
2k
views
How to extract text from websites
I am looking for a way to automate extracting text from multiple websites to a word document. When pasted into the word doc I need it to paste using "merge formatting". This is how I would like the ...
2
votes
1
answer
3k
views
How to allocate different IP while crawling web pages
How can i assign different ip address while crawling
I am crawling billions of page everyday(allocating different ip will help to handle blockages)
1
vote
1
answer
70
views
save web page with all the related content
I am trying to figure how I can save a webpage with all the related files, for example : http://docs.oasis-open.org/ubl/os-UBL-2.0/xsd/
I want to save all the files in the directory kinda like a ...
1
vote
1
answer
2k
views
How could I crawl all the files in file server recursively
There are thousands of files in a file server http://xxxx.com
I tried to crawl it with the tool httrack
It doesn't work, is there any alternative tool can download the whole files recursively base ...
1
vote
2
answers
79
views
Web crawler with converte links option
I would like to crawl a backup site I lost access to. The site is backed up in subdomain.somesite.com while the links on the web page are www.subdomain.com
this leads to the following situation:
...
-2
votes
2
answers
2k
views
How can I scrape only word data from a website? [duplicate]
I want to download all the word content from a particular website. Save the results in MS Word, Excel or Notepad and check which words are repeated most often and how many times.
1
vote
0
answers
170
views
wget - limit following to specific links
I want to do mirror of website but limit following only to specific links.
Command:
wget --recursive --page-requisites --html-extension --convert-links --domains linuxmagazine.pl http://www....
-1
votes
2
answers
381
views
The "smart" way to crawl the web [closed]
I have a project in which I have to crawl the web for lyrics, I am implementing depth-first search and I am making a database in a list (which contains nested lists which will later be saved in a file)...
0
votes
1
answer
159
views
Why is my personal web site getting visitors at mysterious URLs? [closed]
I recently made an update to my site's visitor logging and when checking that it was working properly noticed that I'm getting visitors at URLs that do not belong to me. I was surprised to even see ...
1
vote
0
answers
49
views
Access to all links on a domain(no hyperlink available)
I need to have access to all links like:
http://www.thewebsitename.com/random_alphanumeric_code1.asp
http://www.thewebsitename.com/random_alphanumeric_code2.asp
etc.for a given website: http://...
2
votes
3
answers
1k
views
Extracting links from a numeric range of web pages
I would like to extract links from a numerical sequence of pages like this:
http://example.com/page001.html
http://example.com/page002.html
http://example.com/page003.html
...
http://example.com/...
-1
votes
1
answer
799
views
List all links of one website on other website [closed]
There are two websites.
1) www.aa.com
2) www.bb.com
Now there are so many link of aa.com and its webpages on bb.com
like
aa.com/blhahhaa.html
aa.com/beautifulday.html
aa.com
so i want to find out ...
2
votes
1
answer
2k
views
Mirroring a web site having pages that uses simple JavaScript [duplicate]
There are simple and great web site download tools that allow to create a local mirror of simple sites having no JavaScript infrastructure behind. However, sometimes there cases when a site looks ...
-1
votes
1
answer
557
views
Compiling a list of links on a website and their validity [closed]
I would like to crawl my website and generate a list of internal and outbound links, and what their final destination is and HTTP Code (at least for internal links). How can I do that?
Please note: I ...
1
vote
1
answer
148
views
How would I scrape text from a site? [closed]
Schemer is closing down, and I'm trying to preserve the text because there are many good ideas on the site. I'd like to get the text from within each of the links listed on the sitemap.
Is there a ...
24
votes
2
answers
19k
views
How to crawl using wget to download ONLY HTML files (ignore images, css, js)
Essentially, I want to crawl an entire site with Wget, but I need it to NEVER download other assets (e.g. imagery, CSS, JS, etc.). I only want the HTML files.
Google searches are completely useless.
...
15
votes
2
answers
2k
views
Why is @ in email address sometimes written as [at] on webpages?
Why is @ sometimes in webpages written as [at]? Does it have any specific reason ?
0
votes
1
answer
484
views
Wget getting responce 403
I am using API that has some limit of requests in an hour. But my script does all at one time so I lose about 1/3 of requests because I get 403.
Is there any way to check response of wget and if I ...
2
votes
2
answers
4k
views
Web scraping / crawling a particular Google book
For my work, I need to scrape the text from a large book on Google Books. The book in question is a very old book and is out of copyright. The book is a Gazetteer of the World. We will be putting the ...
1
vote
1
answer
2k
views
How to crawl your own website to save to cache
I'm using Squid, a caching program, to cache my website. However to do so it appears that each page has to be accessed at least once before Squid can cache it. My question is, is there a program which ...
2
votes
1
answer
2k
views
recursively downloading all the folders and subfolder from webpage
I would like to download a directory from webpage. i am trying to do using wget and crawl but i am not able to download the whole directory as it was there in web-page or in any-other way.
suppose the ...
0
votes
1
answer
972
views
Windows - Crawl URL and grab links
I have recently signed up to a large video site as a premium member and I want to generate a list of the video files on the site so that I can put them into my download manager and download the ...
-1
votes
2
answers
4k
views
Streaming Video Bulk Download
There is a video series I have been watching: http://english.cntv.cn/program/learnchinese/growingwithchinese/index.shtml
I have tried a few plugins and utilities to download the episodes with varying ...
4
votes
2
answers
13k
views
How can I scrape specific data from a website
I'm trying to scrape data from a website for research.
The urls are nicely organized in an example.com/x format, with x as an ascending number and all of the pages are structured in the same way. I ...
3
votes
5
answers
16k
views
Website crawler/spider to get site map [closed]
I need to retrieve a whole website map, in a format like :
http://example.org/
http://example.org/product/
http://example.org/service/
http://example.org/about/
http://example.org/product/viewproduct/...