up vote 1 down vote favorite
share [g+] share [fb]

I'm using wget to create a backup of a site that's running locally on my computer. The command I'm using is:

wget -rkE http://localhost/mpmf/www/dev

So this actually works pretty well, but at the top and bottom of each file there are some random-ish characters. For example, the first line of index.html is 4c11 and the last line is just 0. It looks like a lot of the files end with 0 but I haven't checked each one to see if that's the case.

It seems like there might be some reason that wget is doing this. I thought it might have something to do with the way it goes through the site recursively, but I don't know. So is there a way I can do this without these extra lines in my files?

link|improve this question
feedback

2 Answers

up vote 1 down vote

These are chunk markers of HTTP Chunked transfer-encoding as defined in RFC 2616. They are hexadecimal numbers indicating the length of the chunk that follows.

Support for chunked transfer encoding is required in HTTP/1.1. However, Wget only ever makes HTTP/1.0 requests – and it seems that your web server is ignoring the request version and returning a HTTP/1.1 response, in violation of RFC 2145.

link|improve this answer
Thanks for the answer! This is the second site I've done this with, but didn't have problems the first time. Do you have any guess at what could be different? – seth Mar 22 '11 at 18:51
@seth: What HTTP servers are both sites running? – grawity Mar 22 '11 at 18:56
They're both running on the same Apache server. They have different .htaccess files; maybe it has something to do with that? – seth Mar 22 '11 at 18:57
@seth: I could not find more information, but this might be related to the languages the sites are written in (static pages? PHP? Ruby? etc). It seems that Apache2 always returns HTTP/1.1 in reply to 1.0 requests, although in my tests it does use Content-Length, not chunked transfer. – grawity Mar 22 '11 at 19:07
They're both in PHP. Can't really see anything significantly different between them – seth Mar 23 '11 at 19:15
feedback
up vote 1 down vote

Yeah, it's permitted for servers to respond to HTTP/1.0 requests with HTTP/1.1 responses, provided that they avoid HTTP/1.1-specific features (such as chunked encoding).

It's not just the top and bottom of your files, either - at least potentially. You're liable to find those chunk headers peppered throughout the file, too (making for a fairly corrupted file).

Without knowing the details of your server config, I couldn't speak as to why one of your Apache/PHP setups ignores wget's requesting HTTP version, and one respects it.

However, I will note that the current development sources (which will eventually be wget 1.12.1, I think), includes support for HTTP/1.1 requests and chunked content. There are no deb packages available for it, but if you're used to building from sources, that's an option. The latest test source package (easiest to build from, otherwise you have to grab an ungodly set of prerequisites) are at ftp://alpha.gnu.org/gnu/wget/wget-1.12-2460.tar.bz2, and were last updated Mar 16.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.