Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

How do you use wget to download an entire site (domain A) when its resources are on another domain, (domain B)?
I've tried:
wget -r --level=inf -p -k -E --domains=domainA,domainB http://www.domainA

share|improve this question
    
Wow! No one after all this time? –  Parsa Oct 1 '10 at 8:03
    
The reason that command doesn't work is because using --domains by itself doesn't turn --span-hosts on. Adding --span-hosts would've solved the problem. :| –  Parsa Oct 19 at 1:19

3 Answers 3

up vote 5 down vote accepted

wget --recursive --level=inf --page-requisites --convert-links --html-extension --span-hosts=domainA,domainB url-on-domainA

UPDATE: I remember I copy/pasted the above command (That was 2010 and I was using GNU Tools for Windows back then), however I had to change it to the following when I wanted to use it today: wget --recursive --level=inf --page-requisites --convert-links --adjust-extension --span-hosts --domains=domainA,domainB domainA

The shorthand for that would be: wget -rEDpkH -l inf domainA,domainB domainA

share|improve this answer
    
I get: wget: --span-hosts: Invalid boolean domainA,domainB'; use on' or `off'. After changing to on, it does not work. –  Matthew Flaschen Feb 14 at 1:26
    
@MatthewFlaschen What I've written here worked for me. Could you provide the arguments you've used? –  Parsa Feb 26 at 2:04
    
I don't have the exact command I ran before. However, I have the same problem with: wget --recursive --level=inf --page-requisites --convert-links --html-extension --span-hosts=example.org,iana.org example.org I'm using GNU Wget 1.13.4 on Debian. –  Matthew Flaschen Feb 28 at 5:42
1  
Try --span-hosts --domains=example.org,iana.org - I think --span-hosts needs to be a boolean, and then you use --domains to specify which hosts to span. –  Konklone Oct 18 at 20:47
    
Konklone, --span-hosts is a boolean from 1.12 and later, I didn't know that. @MatthewFlaschen, I updated the answer. By the way, that will still work on 1.11 and earlier, if you're using GNU Tools for Windows. –  Parsa Oct 19 at 1:11

wget --recursive --level=inf --page-requisites --convert-links --html-extension -rH -DdomainA,domainB domainA

share|improve this answer
    
This partly works. However, for some reason, it doesn't seem to work if the URL (at the end) is a redirect. Also, it downloads links too, not just page requisites. Also, -r and --recursive are the same. –  Matthew Flaschen Feb 14 at 1:44
wget --page-requisites --convert-links --adjust-extension --span-hosts --domains domainA,domainB domainA

You might need to ignore robots.txt (note, this may be a violation of some terms of service, and you should download the minimum required). See https://www.gnu.org/software/wget/manual/wget.html#Robot-Exclusion .

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.