Sign up ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

How can I download all pages from a website?

Any platform is fine.

share|improve this question
Check out on Server Fault. – Marko Carter Jul 28 '09 at 13:55
@joe: Might help if you'd give details about what the missing features are... – Ilari Kajaste Sep 23 '09 at 11:06

17 Answers 17

up vote 184 down vote accepted

HTTRACK works like a champ for copying the contents of an entire site. This tool can even grab the pieces needed to make a website with active code content work offline. I am amazed at the stuff it can replicate offline.

This program will do all you require of it.

Happy hunting!

share|improve this answer
Been using this for years - highly recommended. – Umber Ferrule Aug 9 '09 at 20:38
Would this copy the actual ASP code that runs on the server though? – SmartMethod Mar 19 '10 at 13:02
@Optimal Solutions: No, that's not possible. You'd need access to the servers or the source code for that. – Sasha Chedygov Mar 31 '10 at 7:08

I'll address the online buffering that browsers use...

Typically most browsers use a browsing cache to keep the files you download from a website around for a bit so that you do not have to download static images and content over and over again. This can speed up things quite a bit under some circumstances. Generally speaking, most browser caches are limited to a fixed size and when it hits that limit, it will delete the oldest files in the cache.

ISPs tend to have caching servers that keep copies of commonly accessed websites like ESPN and CNN. This saves them the trouble of hitting these sites every time someone on their network goes there. This can amount to a significant savings in the amount of duplicated requests to external sites to the ISP.

share|improve this answer

Wget is a classic command-line tool for this kind of task. It comes with most Unix/Linux systems, and you can get it for Windows too. On a Mac, Homebrew is the easiest way to install it (brew install wget).

You'd do something like:

wget -r --no-parent

For more details, see Wget Manual and its examples, or e.g. these:

share|improve this answer
There's no better answer than this - wget can do anything :3 – Phoshi Sep 16 '09 at 22:30
+1, consider adding the mirror option. – nik Sep 19 '09 at 4:30
+1 for including the --no-parent. definitely use --mirror instead of -r. and you might want to include -L/--relative to not follow links to other servers. – quack quixote Oct 9 '09 at 12:43
@optimal, the HTML output of course - it would get the code only if the server was badly misconfigured – Jonik Mar 19 '10 at 15:17
Homebrew shows how to install it right on their homepage – Eric Brotto Jul 14 '14 at 10:43

I have not done this in many years, but there are still a few utilities out there. You might want to try Web Snake. I believe I used it years ago. I remembered the name right away when I read your question.

I agree with Stecy. Please do not hammer their site. Very Bad.

share|improve this answer

Internet Download Manager has a Site Grabber utility with a lot of options - which lets you completely download any website you want, the way you want it.

  1. You can set the limit on the size of the pages/files to download

  2. You can set the number of branch sites to visit

  3. You can change the way scripts/popups/duplicates behave

  4. You can specify a domain, only under that domain all the pages/files meeting the required settings will be downloaded

  5. The links can be converted to offline links for browsing

  6. You have templates which let you choose the above settings for you

enter image description here

The software is not free however - see if it suits your needs, use the evaluation version.

share|improve this answer

DownThemAll is a Firefox add-on that will download all the content (audio or video files, for example) for a particular web page in a single click. This doesn't download the entire site, but this may be sort of thing the question was looking for.

share|improve this answer

Use wget:

wget --mirror -p --html-extension --convert-links

the options explained:

-p                  get all images, etc. needed to display HTML page.  
--mirror            turns on recursion and time-stamping, sets infinite 
                      recursion depth and keeps FTP directory listings
--html-extension    save HTML docs with .html extensions  
--convert-links     make links in downloaded HTML point to local files.  
share|improve this answer
+1 for --mirror – Johan Sep 17 '09 at 7:09
+1 for providing the explanations for the suggested options. (Although I don't think --mirror is very self-explanatory. Here's from the man page: "This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. It is currently equivalent to -r -N -l inf --no-remove-listing") – Ilari Kajaste Sep 23 '09 at 11:04
If you don’t want to download everything into a folder with the name of the domain you want to mirror, create your own folder and use the -nH option (which skips the host part). – Rafael Jan 3 '12 at 15:33
What about if the Auth is required? – Val May 13 '13 at 16:04
I tried using your wget --mirror -p --html-extension --convert-links and it just downloaded the index. I think you need the -r to download the entire site. – Eric Brotto Jul 14 '14 at 10:49

I like Offline Explorer.
It's a shareware, but it's very good and easy to use.

share|improve this answer

Try BackStreet Browser.

It is a free, powerful offline browser. A high-speed, multi-threading website download and viewing program. By making multiple simultaneous server requests, BackStreet Browser can quickly download entire website or part of a site including HTML, graphics, Java Applets, sound and other user definable files, and saves all the files in your hard drive, either in their native format, or as a compressed ZIP file and view offline.

enter image description here

share|improve this answer

Teleport Pro is another free solution that will copy down any and all files from whatever your target is (also has a paid version which will allow you to pull more pages of content).

share|improve this answer

I believe google chrome can do this on desktop devices, just go to the browser menu and click save webpage.

Also note that services like pocket may not actually save the website, and are thus susceptible to link rot.

Lastly note that copying the contents of a website may infringe on copyright, if it applies.

share|improve this answer

For Linux and OS X: I wrote grab-site for archiving entire websites to WARC files. These WARC files can be browsed or extracted. grab-site lets you control which URLs to skip using regular expressions, and these can be changed when the crawl is running. It also comes with an extensive set of defaults for ignoring junk URLs.

There is a web dashboard for monitoring crawls, as well as additional options for skipping video content or responses over a certain size.

share|improve this answer

download HTTracker it will download websites very easy steps to follows.

download link:

video that help may help you :

share|improve this answer

The venerable has this feature too.

Free Download Manager has it in two forms in two forms: Site Explorer and Site Spider:

Site Explorer
Site Explorer lets you view the folders structure of a web site and easily download necessary files or folders.
HTML Spider
You can download whole web pages or even whole web sites with HTML Spider. The tool can be adjusted to download files with specified extensions only.

I find Site Explorer is useful to see which folders to include/exclude before you attempt attempt to download the whole site - especially when there is an entire forum hiding in the site that you don't want to download for example.

share|improve this answer

protected by Community Apr 16 '13 at 10:22

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.