Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

How do you use wget to download an entire site (domain A) when its resources are on another domain, (domain B)?
I've tried:
wget -r --level=inf -p -k -E --domains=domainA,domainB http://www.domainA

share|improve this question
Wow! No one after all this time? – Parsa Oct 1 '10 at 8:03
The reason that command doesn't work is because using --domains by itself doesn't turn --span-hosts on. Adding --span-hosts would've solved the problem. :| – Parsa Oct 19 '14 at 1:19
up vote 6 down vote accepted

wget --recursive --level=inf --page-requisites --convert-links --html-extension --span-hosts=domainA,domainB url-on-domainA

UPDATE: I remember I copy/pasted the above command (That was 2010 and I was using GNU Tools for Windows back then), however I had to change it to the following when I wanted to use it today: wget --recursive --level=inf --page-requisites --convert-links --adjust-extension --span-hosts --domains=domainA,domainB domainA

The shorthand for that would be: wget -rEDpkH -l inf domainA,domainB domainA

share|improve this answer
I get: wget: --span-hosts: Invalid boolean domainA,domainB'; use on' or `off'. After changing to on, it does not work. – Matthew Flaschen Feb 14 '14 at 1:26
@MatthewFlaschen What I've written here worked for me. Could you provide the arguments you've used? – Parsa Feb 26 '14 at 2:04
I don't have the exact command I ran before. However, I have the same problem with: wget --recursive --level=inf --page-requisites --convert-links --html-extension, I'm using GNU Wget 1.13.4 on Debian. – Matthew Flaschen Feb 28 '14 at 5:42
Try --span-hosts, - I think --span-hosts needs to be a boolean, and then you use --domains to specify which hosts to span. – Konklone Oct 18 '14 at 20:47
Konklone, --span-hosts is a boolean from 1.12 and later, I didn't know that. @MatthewFlaschen, I updated the answer. By the way, that will still work on 1.11 and earlier, if you're using GNU Tools for Windows. – Parsa Oct 19 '14 at 1:11

wget --recursive --level=inf --page-requisites --convert-links --html-extension -rH -DdomainA,domainB domainA

share|improve this answer
This partly works. However, for some reason, it doesn't seem to work if the URL (at the end) is a redirect. Also, it downloads links too, not just page requisites. Also, -r and --recursive are the same. – Matthew Flaschen Feb 14 '14 at 1:44
wget --page-requisites --convert-links --adjust-extension --span-hosts --domains domainA,domainB domainA

You might need to ignore robots.txt (note, this may be a violation of some terms of service, and you should download the minimum required). See .

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .