How do you use wget to download an entire site (domain A) when its resources are on another domain, (domain B)?
I've tried:
wget -r --level=inf -p -k -E --domains=domainA,domainB http://www.domainA
wget --recursive --level=inf --page-requisites --convert-links --html-extension \
--span-hosts=domainA,domainB url-on-domainA
UPDATE: I remember the command above worked for me in the past (that was 2010 and I was using GNU Tools for Windows back then); however I had to change it to the following when I wanted to use it today:
wget --recursive --level=inf --page-requisites --convert-links \
--adjust-extension --span-hosts --domains=domainA,domainB domainA
The shorthand for that would be: wget -rEDpkH -l inf domainA,domainB domainA
-r
=--recursive
-l <depth>
=--level=<depth>
-E
=--adjust-extension
-p
=--page-requisites
-K
=--backup-converted
-k
=--convert-links
-D <domain-list>
=--domains=<domain-list>
-H
=--span-hosts
-np
=--no-parent
-U <agent-string>
=--user-agent=<agent-string>
GNU Wget Manual: https://gnu.org/software/wget/manual/wget.html
-
I get: wget: --span-hosts: Invalid boolean
domainA,domainB'; use
on' or `off'. After changing to on, it does not work. – Matthew Flaschen Feb 14 '14 at 1:26 -
@MatthewFlaschen What I've written here worked for me. Could you provide the arguments you've used? – Parsa Feb 26 '14 at 2:04
-
I don't have the exact command I ran before. However, I have the same problem with:
wget --recursive --level=inf --page-requisites --convert-links --html-extension --span-hosts=example.org,iana.org example.org
I'm using GNU Wget 1.13.4 on Debian. – Matthew Flaschen Feb 28 '14 at 5:42 -
3Try
--span-hosts --domains=example.org,iana.org
- I think--span-hosts
needs to be a boolean, and then you use--domains
to specify which hosts to span. – Eric Mill Oct 18 '14 at 20:47 -
Konklone, --span-hosts is a boolean from 1.12 and later, I didn't know that. @MatthewFlaschen, I updated the answer. By the way, that will still work on 1.11 and earlier, if you're using GNU Tools for Windows. – Parsa Oct 19 '14 at 1:11
wget --recursive --level=inf --page-requisites --convert-links --html-extension -rH -DdomainA,domainB domainA
-
This partly works. However, for some reason, it doesn't seem to work if the URL (at the end) is a redirect. Also, it downloads links too, not just page requisites. Also, -r and --recursive are the same. – Matthew Flaschen Feb 14 '14 at 1:44
wget --page-requisites --convert-links --adjust-extension --span-hosts --domains domainA,domainB domainA
You might need to ignore robots.txt (note, this may be a violation of some terms of service, and you should download the minimum required). See https://www.gnu.org/software/wget/manual/wget.html#Robot-Exclusion .
Consider using HTTrack. It has more options when crawling content on other domains than wget. Using wget with --span-hosts, --domains and --accept where insufficient for my needs but HTTrack did the job. I remember that setting limit of re-directions on other domains helped a lot.
--domains
by itself doesn't turn--span-hosts
on. Adding--span-hosts
would've solved the problem. :| – Parsa Oct 19 '14 at 1:19