How do I use HTTrack to download gzipped files from URLs with accented characters?

Question

I am downloading a site with HTTrack with spotty results. Several directories return 2 or more versions of the same HTML file. These duplicates in any given directory may include:

1. a file named índice.html (note the accented í) that shows gibberish in the browser. When studied more carefully, this turns out to be a .z archive with an incorrect extension, containing the correct HTML file
2. a file named índice.html.z, which is a an archive containing a readable version of that file
3. a file named índice-2.html, which is a good version of the original índice.html, perfectly readable in the browser
4. a file named índice-2.html.z, which is an archive containing the same file again, but sometimes that file will be somewhat different in size from the first one
5. etc

The HTTrack error log shows the following:

18:07:32 Error: "Error when decompressing" (-1) at link example.com/conversación/índice.html

This is a Spanish site, and some directories have accents in them and files are called índice.html instead of index.html. This makes me suspect that the reason HTTrack messes up the download is the accents, but I can't prove it, except that I downloaded the English version of the same site without any problems.

To summarize, the problem might lie either in the accented characters in the URL or something else related to HTTrack's way of handling gzipped HTML files, but my main question remains the same:

Is this a bug in HTTrack or expected behavior, and how do I get around it to download the Spanish version of the site successfully?

Answer 1

It may be caused by the accented characters, as you suspect. This open bug appears to be related.

Seems caused by Chinese text in .whtt title. Sometimes the Chinese text will cause Winhttrack to create some rubbish coded folders in the same folder as the .whtt file. And in these cases, there will be .html.z files left in the download folders and sometimes .delayed or empty files and failure with "Error when decompressing" message.

When I retried with plain ASCII project name for the sam URL, it succeeded.

So it seems an encoding problem (which doesn't exist in some earlier versions).

suggestions:

• Try the --utf8-conversion flag.
• If that doesn't work, try downloading an earlier version of the program (since the bug reporter claims some earlier versions didn't exhibit the problem).

• Alternatively, use wget instead. Something like

  wget -mkp -np -nH www.example.com/path/to/toplevel/directory/index.html
  

  will recursively copy all pages linked to index.html which are under the directory example.com/path/to/toplevel/directory/. It also includes any files needed to support those files (CSS, JS, etc).

  note: if wget destroys the accented filenames, use the option

  --restrict-file-names=nocontrol
  

near-duplicate files

As for the issue of near-duplicate files, HTTrack (or wget) can't really do anything about that unless you can tell it how to choose which files it should and shouldn't download.

If there's some kind of consistent naming scheme, and you know you won't want files with a certain kind of name and/or path, you can use a filter to exclude them.

• HTTrack allows files to be excluded using filters. These are wildcard patterns prefixed with -, eg. the filter

  -www.example.com/path/to/toplevel/directory/subdir_with_dupes/*-2.html
  

  will exclude all files in the subdirectory subdir_with_dupes/ that have a name ending in -2.html. There are various wildcards and scan rules that can be used in filters. See the link above, or the man page.

• If you use wget, you can exclude directories with --exclude-directories and filename suffixes with --reject (allowing wildcards). Or you can apply a regular expression filter to the entire URL with --reject-regex. There are lots of other options. You can get a list with wget --help and explanations with man wget.