Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to rip a site via HTTP. I need to download the images, HTML, CSS, and JavaScript as well as organize it in a file system.

Does anyone know how to do this?

share|improve this question

migrated from Apr 13 '10 at 2:39

This question came from our site for professional and enthusiast programmers.

You should mention that you're not going to use this software for copyright infringement. Otherwise we just assume that you want to mirror someone else's content so you can make money from it. – belgariontheking Mar 25 '09 at 19:31
We once had a client want to leave, but take the static HTML version of their site, abandoning the CMS. I used HTTrack to generate the static site. It works well on Windows and Ubuntu. – TRiG Dec 7 '10 at 21:45
@belgariontheking: Interesting you assume the worst. We also use WinHTTrack to download a static HTML version of our own dynamic web site. – Umber Ferrule Aug 3 '11 at 12:40

10 Answers 10

up vote 38 down vote accepted
wget -erobots=off --no-parent --wait=3 --limit-rate=20K -r -p -U "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" -A htm,html,css,js,json,gif,jpeg,jpg,bmp

This runs in the console.

this will grab a site, wait 3 seconds between requests, limit how fast it downloads so it doesn't kill the site, and mask itself in a way that makes it appear to just be a browser so the site doesn't cut you off using an anti-leech mechanism.

Note the -A parameter that indicates a list of the file types you want to download.

You can also use another tag, -D, to indicate a series of domains you want to download if they have another server or whatever for hosting different kinds of files. There's no safe way to automate that for all cases, if you don't get the files.

wget is commonly preinstalled on Linux, but can be trivially compiled for other Unix systems or downloaded easily for Windows:GNUwin32 WGET

Use this for good and not evil.

share|improve this answer
wget -erobots=off --no-parent --wait=3 --limit-rate=50K -r -p -U "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" -k --directory-prefix "C:\rip" --page-requisites -A htm,aspx,php,jsp,asp,zip,png,html,css,js,json,gif,jpeg,jpg,bmp – Chris S Mar 1 '09 at 18:36
Would this work on "Vanity URLs"? I'm worried about my content (OK, songs) written on a social networking site, and I want to backup it locally. The site is "" but there's a vanity URL with my username "". I don't want to download the whole site, just my content! – Avi Jul 6 '09 at 22:34
On Linux, you may want to install wget from sources. The Ubuntu version, for example, does not parse CSS while upstream wget does. – GDR Oct 26 '11 at 15:50
+1 slick [15 chars] – Matt Nov 24 '11 at 14:29

Good, Free Solution: HTTrack

HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility.

It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the "mirrored" website in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads. HTTrack is fully configurable, and has an integrated help system.

share|improve this answer

On Linux systems, 'wget' does this, pretty much.

Its also been ported to several other platforms, as several of the other answers mention.

share|improve this answer
Windows WGet: – VBNight Feb 6 '09 at 21:41
Since it's Linux I would probably just create an Ubuntu VM, run wget, and transfer the files back to my host computer. Seems like that would be faster than porting to Windows. :) (Granted, porting to Windows is probably a good exercise anyway!) – JMD Feb 6 '09 at 21:45
I run debian on my machine. Can i tell wget to retrieve the associated sub dirs/javascript/css/images by giving it just the http root domain? – blackrobot Feb 6 '09 at 21:46
# Download the entire contents of wget -r -l 0 – VBNight Feb 6 '09 at 21:48
Yeah... or use the native Windows port, or use Cygwin... – Tmdean Feb 6 '09 at 21:52

Obviously WGet has been mentioned a few times. The best UI I've found for it is

There's some other UIs for WGet around out, some of which are candidates for the worst ui question

share|improve this answer

Look at the Scrapbook extension for firefox. It does an amazing job at this and also integrate with firebug and allows you to delete elements out of the DOM before you save if you'd like.

share|improve this answer

You need to use wget - which is available for most platforms. curl will not request documents recursively, which is one of wget's major strengths.

Linux: (usually included in the distro)

PLEASE make sure you aren't hammering the website - set up suitable delays between requests, and make sure it's within the site's terms of service.


share|improve this answer
actually, it's our servers that we'd be hammering. the site is dynamically generated via PHP and the CMS/DB was sort of mangled. it's a convoluted story, i'm the developer who just got here. but we're building everything in django now, so you needn't worry. – blackrobot Feb 6 '09 at 21:54

Actually, following up my comment in GWLlosa's post, I just remembered I have GnuWin32 installed, and sure enough it contains a Windows port of wget.

GnuWin32 provides Win32-versions of GNU tools,
or tools with a similar open source licence.
The ports are native ports, that is they rely
only on libraries provided with any 32-bits
MS-Windows operating system, such as
MS-Windows 95 / 98 / 2000 / NT / XP
share|improve this answer

I used this some years ago and it worked well. Windows only. Used to be adware but no longer, apparently:

share|improve this answer

I think IDM site grabber is the best solution, there is also Teleport pro

share|improve this answer

Free Download Manager can download complete web-sites as well.

Windows only I think.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .