1

I'm trying to download site with command:

wget -r -l 0 -p -k -m -e robots=off --restrict-file-names=nocontrol,unix http://example.com

When it finds the link, for example, http://example.com/direction, it creates a file on the disk ./example.com/direction

Then, for example, it finds some other files (links) in 'direction', suppose http://example.com/direction/11

And wget rewrite old file ./example.com/direction with some HTML, and transform it into the directory. And write new file ./example.com/direction/11 there.

If it finds a new file inside /direction/11, then it'll rewrite 11 as well and convert it into the directory.

So, it seems, I've lost all directory index files.

Is there a solution for the problem in wget itself?

2

This may not work in every case, but I was able to solve this problem using a somewhat unexpected tag.

From the wget 1.17.1 manual:

‘-E’

‘--adjust-extension’

If a file of type ‘application/xhtml+xml’ or ‘text/html’ is downloaded and the URL does not end with the regexp ‘\.[Hh][Tt][Mm][Ll]?’, this option will cause the suffix ‘.html’ to be appended to the local filename.

Thus, in your case, when wget finds the link http://example.com/direction, it will check whether the page is html-formatted. If so, it will save it to the local disk as /example.com/direction.html. Then when it sees http://example.com/direction/11, it will save it to the disk as /example.com/direction/11, thus not overwriting the first file. Assuming that the true directory structure is /example.com/direction/index.html, it won't exactly match the remote directory structure, but at least it won't incorrectly overwrite files.

0

Well, a solution to this issue using only Wget is nigh impossible. Let me first try and explain the issue to you.

First Wget attempts to download, http://site.com/direction. When Wget sends a request for this path, the server responds with the data available in http://site/com/example/index.html. However, at this point, Wget is not aware that the data being sent to it corresponds to the path: example/index.html. Now, when Wget finds another file, http://site.com/example/11, the same things happens again. However, it must first create the directory example/ and it unconditionally deletes the old file to do this. IMO, Wget should detect if it is replacing a file with a directory and save the file as an index file. But, as of now it doesn't.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.