16

How do you use wget to download an entire site (domain A) when its resources are on another domain, (domain B)?
I've tried:
wget -r --level=inf -p -k -E --domains=domainA,domainB http://www.domainA

|improve this question
  • Wow! No one after all this time? – Parsa Oct 1 '10 at 8:03
  • The reason that command doesn't work is because using --domains by itself doesn't turn --span-hosts on. Adding --span-hosts would've solved the problem. :| – Parsa Oct 19 '14 at 1:19
14
wget --recursive --level=inf --page-requisites --convert-links --html-extension \
     --span-hosts=domainA,domainB url-on-domainA

UPDATE: I remember the command above worked for me in the past (that was 2010 and I was using GNU Tools for Windows back then); however I had to change it to the following when I wanted to use it today:

wget --recursive --level=inf --page-requisites --convert-links \
     --adjust-extension --span-hosts --domains=domainA,domainB domainA

The shorthand for that would be: wget -rEDpkH -l inf domainA,domainB domainA

  • -r = --recursive
  • -l <depth> = --level=<depth>
  • -E = --adjust-extension
  • -p = --page-requisites
  • -K = --backup-converted
  • -k = --convert-links
  • -D <domain-list> = --domain-list=<domain-list>
  • -H = --span-hosts
  • -np = --no-parent
  • -U <agent-string> = --user-agent=<agent-string>

GNU Wget Manual: https://www.gnu.org/software/wget/manual/wget.html

|improve this answer
  • I get: wget: --span-hosts: Invalid boolean domainA,domainB'; use on' or `off'. After changing to on, it does not work. – Matthew Flaschen Feb 14 '14 at 1:26
  • @MatthewFlaschen What I've written here worked for me. Could you provide the arguments you've used? – Parsa Feb 26 '14 at 2:04
  • I don't have the exact command I ran before. However, I have the same problem with: wget --recursive --level=inf --page-requisites --convert-links --html-extension --span-hosts=example.org,iana.org example.org I'm using GNU Wget 1.13.4 on Debian. – Matthew Flaschen Feb 28 '14 at 5:42
  • 3
    Try --span-hosts --domains=example.org,iana.org - I think --span-hosts needs to be a boolean, and then you use --domains to specify which hosts to span. – Eric Mill Oct 18 '14 at 20:47
  • Konklone, --span-hosts is a boolean from 1.12 and later, I didn't know that. @MatthewFlaschen, I updated the answer. By the way, that will still work on 1.11 and earlier, if you're using GNU Tools for Windows. – Parsa Oct 19 '14 at 1:11
1

wget --recursive --level=inf --page-requisites --convert-links --html-extension -rH -DdomainA,domainB domainA

|improve this answer
  • This partly works. However, for some reason, it doesn't seem to work if the URL (at the end) is a redirect. Also, it downloads links too, not just page requisites. Also, -r and --recursive are the same. – Matthew Flaschen Feb 14 '14 at 1:44
0
wget --page-requisites --convert-links --adjust-extension --span-hosts --domains domainA,domainB domainA

You might need to ignore robots.txt (note, this may be a violation of some terms of service, and you should download the minimum required). See https://www.gnu.org/software/wget/manual/wget.html#Robot-Exclusion .

|improve this answer
-1

Consider using HTTrack. It has more options when crawling content on other domains than wget. Using wget with --span-hosts, --domains and --accept where insufficient for my needs but HTTrack did the job. I remember that setting limit of re-directions on other domains helped a lot.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.