85

I want to use Wget to save single web pages (not recursively, not whole sites) for reference. Much like Firefox's "Web Page, complete".

My first problem is: I can't get Wget to save background images specified in the CSS. Even if it did save the background image files I don't think --convert-links would convert the background-image URLs in the CSS file to point to the locally saved background images. Firefox has the same problem.

My second problem is: If there are images on the page I want to save that are hosted on another server (like ads) these wont be included. --span-hosts doesn't seem to solve that problem with the line below.

I'm using: wget --no-parent --timestamping --convert-links --page-requisites --no-directories --no-host-directories -erobots=off http://domain.tld/webpage.html

1
  • 1
    exactly the same line ( wget --no-parent --timestamping --convert-links --page-requisites --no-directories --no-host-directories -erobots=off domain.tld ) actually saves background images referenced from CSS after updating to 1.12. The manual says: "With http urls, Wget retrieves and parses the html or css from the given url, retrieving the files the document refers to, through markup like href or src, or css uri values specified using the ‘url()’ functional notation." Second problem still needs to be solved
    – user14124
    Oct 14, 2009 at 0:23

4 Answers 4

124

From the Wget man page:

Actually, to download a single page and all its requisites (even if they exist on separate websites), and make sure the lot displays properly locally, this author likes to use a few options in addition to ‘-p’:

wget -E -H -k -K -p http://www.example.com/

Also in case robots.txt is disallowing you add -e robots=off

6
  • It seems that it is just rewriting js and css to absolute urls
    – Greg Dean
    Sep 30, 2012 at 6:53
  • 1
    nevermind, it was robots.txt disallowing me update the answer with the workaround
    – Greg Dean
    Sep 30, 2012 at 7:12
  • 25
    Expanded: wget --adjust-extension --span-hosts --convert-links --backup-converted --page-requisites [url]
    – sam
    Aug 16, 2013 at 16:25
  • 4
    Or better yet wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows [url]
    – Petah
    Oct 22, 2014 at 3:34
  • 3
    For a page I was trying to get it worked 100% with command wget -E -H -k -K -p -e robots=off URL, thank you.
    – lowtechsun
    Feb 23, 2019 at 11:31
7

The wget command offers the option --mirror, which does the same thing as:

$ wget -r -N -l inf --no-remove-listing

You can also throw in -x to create a whole directory hierarchy for the site, including the hostname.

You might not have been able to find this if you aren't using the newest version of wget however.

1
  • 2
    This will likely crawl the whole website with its sub-urls Sep 13, 2018 at 2:14
2

It sounds like wget and Firefox are not parsing the CSS for links to include those files in the download. You could work around those limitations by wget'ing what you can, and scripting the link extraction from any CSS or Javascript in the downloaded files to generate a list of files you missed. Then a second run of wget on that list of links could grab whatever was missed (use the -i flag to specify a file listing URLs).

If you like Perl, there's a CSS::Parser module on CPAN that may give you an easy means to extract links in this fashion.

Note that wget is only parsing certain html markup (href/src) and css uris (url()) to determine what page requisites to get. You might try using Firefox addons like DOM Inspector or Firebug to figure out if the 3rd-party images you aren't getting are being added through Javascript -- if so, you'll need to resort to a script or Firefox plugin to get them too.

3
  • Like I said in the comment for my first post it seems it has been fixed in v1.12. I still don't know how to include images that are on other servers though.
    – user14124
    Oct 14, 2009 at 0:32
  • yep, parsing the CSS is new in wget v1.12, it's at the top of the changelog: freshmeat.net/urls/376000c9c7a02f7a3592180c2390ff04 Oct 14, 2009 at 0:41
  • Try adding the option -H to the list. It stands for --span-hosts and allows downloading of content from external hosts. Dec 30, 2009 at 21:47
2

I made Webtography for a similar purpose: https://webjay.github.io/webtography/

It uses Wget and pushes the site to a repository on your GitHub account.

I use these arguments:

--user-agent=Webtography
--no-cookies
--timestamping
--recursive
--level=1
--convert-links
--no-parent
--page-requisites
--adjust-extension
--max-redirect=0
--exclude-directories=blog

https://github.com/webjay/webtography/blob/master/lib/wget.js#L15-L26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .