1

I am trying to download an entire website for a project. I'm working on Linux and usually use this wget command to do the job

wget -r -p -k -E www.website.org

But for this particular website, not all images were downloaded.

I after some researching I also tried some other options, e.g.:

wget -r -p -k -l 0 -E --no-check-certificate www.website.org

But images are still missing. By inspecting the source code, I noticed that many of the missing images are requested by inline javascript. For example:

<a href="index.php" onMouseOut="MM_swapImgRestore();"  
onMouseOver="MM_swapImage('button','','images/button_highlight.gif',1)" >
   <img name="button" src="images/button.gif">
</a>

Is there a way to include those images with wget?

0

Not really, that's not a thing wget was build for (neither curl, for example). Javascript might even delay loading images by a certain amount of time, or wait for images to appear on screen (cf. long scrolling pages).

In your case, you could programmatically parse the javascript code to extract file names (like images/button.gif), then load them with more calls to wget. But of course that is no general solution to that problem. I have also seen web pages with embedded javascript which do compute image URLs/file names during runtime - per session!

A viable approach to circumvent these javascript obfuscation measures would be to loading the web page into a standard browser, but through a local proxy (squid comes to my mind); and then to examine (save) the proxy cache.

  • Thanks, at least I won't waste my time trying to make it work. I will just parse the html for the files and manually download with wget. – josh21 Mar 18 at 19:43
  • Actually, I just parsed the HTML for the image file URLs and saved them in a images.txt, then downloaded them with wget -i images.txt – josh21 Mar 18 at 20:00
0

I'm not sure how you're site is sturctured but indeed wget cand do that, there are tons of examples to do that in the manual. There is a little note about --requisite, so that would depends on the structure of your website. I never had suck problem so can hardly point to the "fix".

Usually I do something like that :

wget \
     --recursive \
     --no-clobber \
     --page-requisites \
     --adjust-extension \
     --convert-links \
     --restrict-file-names=unix \
     --domains <DOMAIN> \
     --no-parent \
     <URL>

You could also try with --mirror :

wget \
     --mirror \
     --no-clobber \
     --page-requisites \
     --adjust-extension \
     --convert-links \
     --restrict-file-names=unix \
     --domains <DOMAIN> \
     --no-parent \
     <URL>

A an ending note, if you have some files where the name is in "Content-Disposable" headers. (domaine.com/something_that_returns_a_file.php for example). You can try to use those if supported by your version. You can use them toghether.

--content-disposition
--metalink-over-http
--trust-server-names

If there are redirections that give you some troubles like bring you back to the home page and then you start download-ception, you can set the --max-redirect=0.

The manual note is a bit long to put here entirely but it starts like that :

Ordinarily, when downloading a single HTML page, any requisite documents that may be needed to display it properly are not downloaded. Using -r together with -l can help, but since Wget does not ordinarily distinguish between external and inlined documents, one is generally left with "leaf documents" that are missing their requisites.

For instance, say document 1.html contains an "" tag referencing 1.gif and an "" tag pointing to external document 2.html. Say that 2.html is similar but that its image is 2.gif and it links to 3.html. Say this continues up to some arbitrarily high number.

  • There is an example of how the references to the missing images are structured. It's onMouseOver calls that request the files. The suggested command options are not that different from what I use, I also tried --mirror (did not download the files either) – josh21 Mar 20 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.