0
votes
1answer
61 views

Downloading entire web pages within domain [duplicate]

My computer in on LAN. I'm connected to the local domain at MyDomain. Within that domain exists something like WIKI for a training (http://training-wiki/). I would like to download entire WIKI into my ...
0
votes
1answer
33 views

use wget to output everything retrieved into a single txt file

Is it possible to do something like this? wget --recursive --output-document=theEntirePagestuffedIntoATextfile.txt www.example.com And in words what it should do is: "download the entire website ...
1
vote
0answers
323 views

Create a complete local copy of a website using Wget

OK, here is what I need : I'm on Mac (Mac OS X 10.6.8) I want to completely mirror a website on my hard drive (this is what I'm using as a test) I want all images and prerequisites there, so that ...
1
vote
1answer
190 views

How to mirror a website on linux?

There is this abandoned blog witch can disappear any moment, and I want to make a browsable copy this site. The saved pages should contain correct on-page links, liked images and other files like ...
0
votes
5answers
4k views

Website crawler/spider to get site map [closed]

I need to retrieve a whole website map, in a format like : http://example.org/ http://example.org/product/ http://example.org/service/ http://example.org/about/ ...
0
votes
1answer
526 views

Using wget on a site with cgi

I need to mirror a website. However, each of the links on the site's webpage is actually a 'submit' to a cgi script that shows up the resulting page. AFAIK wget should fail on this since it needs ...
0
votes
1answer
2k views

How do you find a size of a web-site? [closed]

Suppose I want to download some wiki site. It looks like it has 2000 articles only - but with wiki technology - they keep several versions of each article... So it still could be terabytes! So my ...
0
votes
0answers
155 views

web spidering/crawling, can i do it or just search engines?

i already had a question answered about web-scraping with wget. but as i read a little more, i realize i may be looking for a web-crawling program. particularly the part about web-crawlers being able ...
5
votes
3answers
2k views

wget not converting links

I am trying to mirror a fairly large site (20,000+ pages) prior to a major overhaul. Basically, I need a backup before cutting over to the new one in case we forgot something we need (we'll have ...