13

For a long time I have assumed that it is not really possible to download a corrupted file via http as long as it's not corrupted on the server and the implementation of http protocol is correct, which is most likely the case for modern mainstream software.

So I always chuckled when I saw a download site offer an md5 hash of a file they provide for download. I haven't seen a case before, where I download a file, the size is correct but the content is not.

Well, today, I had a first case of this. I downloaded an iso of Ubuntu, tried to install it, it failed and after a long research (I just could not believe that the reason could be a corrupted download) I checked the MD5 and what do you know, it was wrong (size was correct). So I re-downloaded it and got yet another wrong md5. Only on my third download the md5 was correct.

So my question is, is it possible in principle to get corrupted download over http, assuming that the implementation is correct, the transfer has finished successfully and that the file is correct on the server. If this is possible, then how can this happen?

Improve this question
1

2 Answers 2

Reset to default
10

Yes, it's possible, especially on poor quality Internet connections – usually wireless, but some wired connections (such as the one I have) also have high error rates at high speeds.

The HTTP protocol does not have any provisions for ensuring data integrity. On transport layer, TCP does have error detection by using a checksum, but it's not very reliable.

There is another reason for providing hashes or digital signatures. Often, the actual files are distributed over many mirror servers, which cannot be guaranteed to be 100% secure. If there's no hash or signature to verify, someone with access to a mirror (not necessarily legitimate) could replace the files and remain undetected, without having to break into a completely different server where the website is hosted.

You can get automatic verification of files if you download Ubuntu over BitTorrent instead of HTTP. (Each piece is verified at download time, so you never have to re-download the entire thing.)

Improve this answer
11
  • 1
    Good answer. However I would like to explore the topic a bit more, if you don't mind. The "high speed" part turned out to be very relevant. I never had any problems at home, but this Ubuntu ISO downloads that I did was at work an the speed was about 5Mb/s. It's hard for me to accept that TCP is not reliable, because almost everything around is based on TCP. Is there anything else but this wikipedia article about TCP unreliability? Do you know exactly how high speed affects the issue? Thank you in advance.
    – Andrew Savinykh
    May 16, 2011 at 1:05
  • 1
    @zespri: TCP is supposed to be reliable, but the checksum cannot catch 100% of all errors. High speed is only a factor with certain connection types, or over unreliable links... Which unfortunately can occur anywhere between you and the server, not necessarily immediately at your end. (I'd test downloading from mirrors located at different countries.) There's of course also the possibility that your file was already corrupted on the server's filesystem...
    – u1686_grawity
    May 16, 2011 at 4:14
  • 1
    if there is a small probability of corruption, then increasing the number of hops you have in between you and the server would increase the chance of corruption???
    – Trevor Boyd Smith
    Sep 14, 2011 at 14:07
  • 1
    @Trevor: Depends on the link quality. Ten hops over Ethernet are far more reliable than a single hop over WiFi. (The routers/switches themselves almost never corrupt data; it's usually the connection that does it.)
    – u1686_grawity
    Sep 14, 2011 at 14:13
  • 1
    @Boris: They're still possible if a) the actual file, on the server, has been corrupted before the HTTPS transfer (e.g. disk or RAM problems on the server), or b) it was corrupted after receiving (disk or RAM problems on the client). Those are mostly outside the scope of HTTP or TLS, though.
    – u1686_grawity
    Sep 2, 2021 at 14:33
1

As Grawity said, it is possible, but in addition, what I have noticed:

Even with stable internet connections, it is possible for a download to finish early without any sort of valid reason - it can just happen.

And most importantly, if you have bad memory, it is possible that the download caches in memory before being written to disk, and as the memory is bad, it is writing the file incorrectly. If you constantly have bad downloads, this is a high posibility.

Improve this answer
3
  • Oh, I hate webservers that randomly cut off downloads at ~50% without any support for resuming...
    – u1686_grawity
    May 14, 2011 at 14:43
  • 1
    Wil: Yep, cut offs are often, and because of this I specifically excluded them from my question, saying that I assume that the sizes are exactly right. Thank you for you comment.
    – Andrew Savinykh
    May 16, 2011 at 1:08
  • Yep that is a key statement for this question "the sizes are exactly right"
    – Trevor Boyd Smith
    Sep 14, 2011 at 14:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .