up vote 39 down vote favorite
28
share [g+] share [fb]

How can I download all pages from a website?

Any platform is fine.

link|improve this question
Check out serverfault.com/questions/45096/website-backup-and-download on Server Fault. – Marko Carter Jul 28 '09 at 13:55
@tnorthcutt, I'm surprised too. If I don't recall awfully wrong, my Wget answer used to be the accepted one, and this looked like a settled thing. I'm not complaining though — all of a sudden the renewed attention gave me more than the bounty's worth of rep. :P – Jonik Sep 17 '09 at 6:05
did you try IDM? superuser.com/questions/14403/… my post is buried down. What did you find missing in IDM? – Lazer Sep 21 '09 at 10:30
3  
@joe: Might help if you'd give details about what the missing features are... – Ilari Kajaste Sep 23 '09 at 11:06
feedback

14 Answers

up vote 53 down vote accepted
+100

http://www.httrack.com/

HTTRACK works like a champ for copying the contents of an entire site. This tool can even grab the pieces needed to make a website with active code content work offline. I am amazed at the stuff it can replicate offline.

This program will do all you require of it.

Happy hunting!

link|improve this answer
Beat me to it by 16 seconds! – Marko Carter Jul 28 '09 at 13:56
1  
Been using this for years - highly recommended. – Umber Ferrule Aug 9 '09 at 20:38
You can also limit the speed of download so you don't use too much bandwidth to the detriment of everyone else. – Umber Ferrule Aug 21 '09 at 22:18
Finally this one is bit better than others ;) – joe Sep 23 '09 at 13:33
1  
@Optimal Solutions: No, that's not possible. You'd need access to the servers or the source code for that. – musicfreak Mar 31 '10 at 7:08
show 1 more comment
feedback
up vote 61 down vote

Wget is a classic command-line tool for this kind of task. It comes with most Unix/Linux systems, and you can get it for Windows too.

You'd do something like: 

wget -r --no-parent http://site.com/songs/

For more details, see Wget Manual and its examples, or e.g. these:

link|improve this answer
2  
There's no better answer than this - wget can do anything :3 – Phoshi Sep 16 '09 at 22:30
2  
+1, consider adding the mirror option. – nik Sep 19 '09 at 4:30
1  
+1 for including the --no-parent. definitely use --mirror instead of -r. and you might want to include -L/--relative to not follow links to other servers. – quack quixote Oct 9 '09 at 12:43
I don't think I've used --mirror myself so I didn't put it the answer. (And it's not really fully "self explanatory" like Paul's answer says...) If you want to elaborate on why it's better than -r I'd appreciate it! – Jonik Oct 9 '09 at 13:06
3  
@optimal, the HTML output of course - it would get the code only if the server was badly misconfigured – Jonik Mar 19 '10 at 15:17
show 2 more comments
feedback
up vote 18 down vote

Use wget:

wget --mirror -p --html-extension --convert-links www.example.com

the options explained: 

-p                  get all images, etc. needed to display HTML page.  
--mirror            self explanatory  
--html-extension    save HTML docs with .html extensions  
--convert-links     make links in downloaded HTML point to local files.
link|improve this answer
1  
+1 for --mirror – Johan Sep 17 '09 at 7:09
2  
+1 for providing the explanations for the suggested options. (Although I don't think --mirror is very self-explanatory. Here's from the man page: "This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. It is currently equivalent to -r -N -l inf --no-remove-listing") – Ilari Kajaste Sep 23 '09 at 11:04
1  
If you don’t want to download everything into a folder with the name of the domain you want to mirror, create your own folder and use the -nH option (which skips the host part). – Rafael Jan 3 at 15:33
feedback
up vote 12 down vote

Just be sure to not hammer the site you want to copy.

If a suggested software has an option to throttle the transfer then use it.

Furthermore, if scheduling is available then schedule the transfer during off-peak hours.

link|improve this answer
6  
Throttling is a good practice. Some websites do not appreciate you grabbing the content like that. – Axxmasterr Jul 28 '09 at 14:04
I wonder if site owners are more worried about some heavy load, or about their content being downloaded. – Arjan Sep 16 '09 at 22:10
wget does have options for this – reinierpost Sep 17 '09 at 10:10
@Arjan: if they didn't want you to download it, why do they give you access?</pedantic-and-silly> – SamB Apr 28 '10 at 19:06
@SamB, I think that's the wrong mindset if we want to keep the internet a good source of quality information... – Arjan Apr 28 '10 at 19:13
show 2 more comments
feedback
up vote 3 down vote

I'll address the online buffering that browsers use...

Typically most browsers use a browsing cache to keep the files you download from a website around for a bit so that you do not have to download static images and content over and over again. This can speed up things quite a bit under some circumstances. Generally speaking, most browser caches are limited to a fixed size and when it hits that limit, it will delete the oldest files in the cache.

ISPs tend to have caching servers that keep copies of commonly accessed websites like ESPN and CNN. This saves them the trouble of hitting these sites every time someone on their network goes there. This can amount to a significant savings in the amount of duplicated requests to external sites to the ISP.

link|improve this answer
feedback
up vote 2 down vote

I have not done this is many years, but there are still a few utilities out there. You might try Web Snake: http://www.websnake.com/web.snake.htm I believe I used it years ago. I remembered the name right away when I read your question.

I agree with Stecy. Please do not hammer their site. Very Bad.

link|improve this answer
feedback
up vote 2 down vote

itsucks - that's the name of the program!

link|improve this answer
feedback
up vote 1 down vote

Here is a Firefox add-on that will download all the content (audio or video files, for example) for a particular web page in a single click. This doesn't download the entire site, but this may be sort of thing the question was looking for.

link|improve this answer
feedback
up vote 1 down vote

I like Offline Explorer.
It's a shareware, but it's very good and easy to use.

link|improve this answer
feedback
up vote 1 down vote

You should take a look at ScrapBook, a Firefox extension. It has an in-depth capture mode.

enter image description here

link|improve this answer
feedback
up vote 1 down vote

Internet Download Manager has a Site Grabber utility with a lot of options - which lets you completely download any website you want, the way you want it.

  1. You can set the limit on the size of the pages/files to download

  2. You can set the number of branch sites to visit

  3. You can change the way scripts/popups/duplicates behave

  4. You can specify a domain, only under that domain all the pages/files meeting the required settings will be downloaded

  5. The links can be converted to offline links for browsing

  6. You have templates which let you choose the above settings for you

enter image description here

The software is not free however - see if it suits your needs, use the evaluation version.

link|improve this answer
feedback
up vote 0 down vote

I just used 

wget -r --no-parent http://site.com/songs/

worked great.

link|improve this answer
feedback
up vote 0 down vote

Try BackStreet Browser.

It is a free, powerful offline browser. A high-speed, multi-threading website download and viewing program. By making multiple simultaneous server requests, BackStreet Browser can quickly download entire website or part of a site including HTML, graphics, Java Applets, sound and other user definable files, and saves all the files in your hard drive, either in their native format, or as a compressed ZIP file and view offline.

enter image description here

link|improve this answer
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.