What is the value of MD5 checksums if the MD5 hash itself could potentially also have been manipulated?

Question

Downloads on websites sometimes have an MD5 checksum, allowing people to confirm the integrity of the file. I have heard this is to allow not only corrupted files to be instantly identified before they cause a problem but also for for any malicious changes to be easily detected.

I follow the logic as far as file corruption is concerned but if someone deliberately wants to upload a malicious file, then they could generate a corresponding MD5 checksum and post that on the download site along with the altered file. This would deceive anyone downloading the file into thinking it was unaltered.

How can MD5 checksums provide any protection against deliberately altered files if there is no way of knowing if the checksum itself has been compromised?

Answer 1

I have heard this is to allow [...] for any malicious changes to be detected also.

Well you heard wrong, then. MD5 (or SHA or whatever) checksums are provided (next to downloads links, specifically) only for verifying a correct download. The only thing they aim to guarantee is that you have the same file as the server. Nothing more, nothing less. If the server is compromised, you’re SOL. It’s really as simple as that.

Answer 2

The solution used by some package management systems such as dpkg is to sign the hash: use the hash as input to one of the public key signing algorithms. See http://www.pgpi.org/doc/pgpintro/#p12

If you have the public key of the signatory, you can verify the signature, which proves the hash is unmodified. This just leaves you with the problem of getting the right public key in advance, although if someone once tampers with the key distribution they also have to tamper with everything you might verify with it otherwise you'll spot that something strange is going on.

Answer 3

Your assumption is correct. There is an exception though. If the server providing the file and the page where the hash is are not managed by the same entity. In that case the software developer may want to say "hey people download this from that place but only believe if hash = xxxx". (This might be usefull for CDN's as an example). I guess this was the reason why someone did it in the first place. Than others just followed thinking how cool it would be to show the hash. Not even thinking how useful it is not even both the file and the hash are on the same location.

Having this said, this is worth what it is. Don't assume too much about security as others already stated. If and only if you can absolutely trust the original hash, than the file is good. Otherwise an attacker with enough motivation and knowledge can tamper both file and the hash, even if these are in different servers and managed by different entities.

Answer 4

Sometimes the checksums are provided securely, but the download is not. Since MD5 is broken, the security MD5 checksums provide are weaker than more secure checksums, but before MD5 was broken, a securely provided MD5 (e.g. one that was signed with PGP or GPG or Gatekeeper, or fetched over HTTPS) that matched the MD5 of the download was strong evidence that the download received was the one the server was making available.

I have been writing about the lamentable lack of secure checksums for years, here.

Users shouldn't download untrusted executables over untrusted networks and run them, because of the risk of MITM attacks. See, e.g. "Insecurities within automatic update systems" by P. Ruissen, R. Vloothuis.

2014 Addendum: No, it's NOT wrong "that checksums posted on web pages are used to detect malicious modifications," because this IS a role they can perform. They do help protect against accidental corruption, and if served over HTTPS or with a verified signature (or better yet, both) help protect against malicious corruption! I have obtained checksums over HTTPS and verified that they matched HTTP downloads many times.

Nowadays, binaries are often distributed with signed, automatically verified hashes, yet even this is not perfectly secure.

Excerpt from above link: "The KeRanger application was signed with a valid Mac app development certificate; therefore, it was able to bypass Apple’s Gatekeeper protection." ... "Apple has since revoked the abused certificate and updated XProtect antivirus signature, and Transmission Project has removed the malicious installers from its website. Palo Alto Networks has also updated URL filtering and Threat Prevention to stop KeRanger from impacting systems. Technical Analysis

The two KeRanger infected Transmission installers were signed with a legitimate certificate issued by Apple. The developer listed this certificate is a Turkish company with the ID Z7276PX673, which was different from the developer ID used to sign previous versions of the Transmission installer. In the code signing information, we found that these installers were generated and signed on the morning of March 4."

2016 Addenda:

@Cornstalks: Re. your comment below: Wrong. As currently noted at the collision attack Wikipedia article you link to, "In 2007, a chosen-prefix collision attack was found against MD5" and "the attacker can choose two arbitrarily different documents, and then append different calculated values that result in the whole documents having an equal hash value." Thus, even if the MD5 is provided securely and an attacker can't modify it, an attacker still CAN use a chosen-prefix collision attack with a chosen-prefix containing malware, which means MD5 is NOT secure for crypto purposes. This is largely why US-CERT said MD5 "should be considered cryptographically broken and unsuitable for further use."

A couple more things: CRC32 is a checksum. MD5, SHA, etc. are more than checksums; they're intended to be secure hashes. That means they're supposed to be very resistant to collision attacks. Unlike a checksum, a securely communicated secure hash protects against a man-in-the-middle (MITM) attack where the MITM is between the server and the user. It doesn't protect against an attack where the server itself is compromised. To protect against that, people typically rely on something like PGP, GPG, Gatekeeper, etc.

Answer 5

This is the precise reason posted checksums often carry a disclaimer saying "This cannot protect against malicious modification of the file". So, the short answer is "they can't provide any protection whatsoever against a deliberately altered file" (although, if the page is delivered over HTTPS, HTTPS itself protects against modification; if the file isn't delivered over HTTPS but the checksum is, then that might help some, but isn't a common case). Whoever told you that checksums posted on web pages are used to detect malicious modifications was wrong, because this is not a role they can perform; all they do is help protect against accidental corruption, and lazy malicious corruption (if someone doesn't bother to intercept the page giving you the checksum).

If you want to protect against deliberate modification, you need to either keep people from messing with the checksum, or make it impossible for anyone else to generate a valid checksum. The former can involve giving it out in person or similar (so the checksum itself is trusted); the latter goes to digital signature algorithms (where you need to securely get your public key to the downloader; in TLS, this is done by ultimately trusting certificate authorities directly and having them verify everyone else; it can also be done over a web of trust, but the point is that something has to be securely transferred at some point, and just posting something on your site isn't enough).

Answer 6

This is really a problem. Showing checksums on the same site as the file to download is insecure. A person who can change the file can also change the checksum. The checksum should be shown through a complete separated system but this is hardly feasible, because how to tell the user in a safe way where the checksum can be found.

A possible solution is the use of signed files.

(BTW: MD5 is unsafe anywhere and shouldn't be used anymore.)

Answer 7

How can MD5 checksums provide any protection against deliberately altered files if there is no way of knowing if the checksum itself has been compromised?

You are entirely correct. The goal, then, would be to make your "if" wrong — if we know that a secure cryptographic hash of a file isn't compromised, then we know that the file isn't compromised either.

For example, if you post a hash of a file on your website, and then link to a copy of the file on a third-party mirror server — common practice in old-fashioned free software distribution — your users can be protected against some types of attacks. If the mirror server is malicious or compromised, but your website is okay, the mirror won't be able to subvert your file.

If your website uses HTTPS, or you sign the hash with gpg, your file can also be (mostly) protected from network attackers like malicious Wi-Fi hotspots, rogue Tor exit nodes, or NSA.

Answer 8

You can't modify the MD5 checksum without also modifying the file. If you download the file, then download the hash, and then your computation of the file's has doesn't match what is given, either the hash or the file is wrong or incomplete.

If you want to "tie" the file to something external, such as author, machine, etc. it needs to be signed, using a PKI-type process with certificates. The file author, etc. can sign the file with his/her private key, and you can verify signatures with the public key, which should be publicly available, itself signed by a CA both you and the author trust, and downloadable, preferably from multiple locations.

Modifying the file would make the signature invalid, so this can be used to verify file integrity too.

Answer 9

Hashes indicate whether your version of the file (the "download") differs from the server's version. They offer no guarantee to the authenticity of the file.

Digital signatures (asymmetric encryption + hash function) can be used to verify that the file has not been modified by anyone who does not have the corresponding private key.

The file's creator hashes the file and encrypts the hash using their (secret) private key. That way, anyone with the corresponding (non-secret) public key can verify that the hash matches the file, but while the file contents can be modified, nobody can replace the corresponding hash with one that matches the file (after hash is decrypted using the public key) - unless they manage to brute-force the private key, or gain access to it somehow.

What's to stop Mr "A.Hacker" from simply modifying the file, then signing it with their own private key?

In order to validate the file, you need to compare its hash to the one that you obtained by decrypting the associated digital signature. If you think the file is from "I.M.Awesome", then you decrypt the hash using his key and the hash does not match the file, since the hash was encrypted using A.Hacker's key.

Digital signatures hence allow one to detect both accidental and malicious changes.

But how do we get I.M.Awesome's public key in the first place? How can we ensure that when we obtained his/her key, it wasn't actually A.Hacker's key being served by a compromised server or a man-in-the-middle attack? This is where certificate chains and trusted root certificates come in, neither of which are perfectly secure [1] solutions to the problem, and both of which should probably be well explained on Wikipedia and on other questions on SO.

[1] Upon inspection, the root certificates that shipped with the Microsoft OS on my work PC include a certificate from the U.S. Government. Anyone with access to the corresponding private key cough NSA cough can therefore serve content to my web browser that passes the "is there a padlock in the address bar" check. How many people will actually bother to click on the padlock and see who's key-pair is being used to "secure" the connection?

Answer 10

How can MD5 checksums provide any protection against deliberately altered files if there is no way of knowing if the checksum has not been compromised either?

This is a really good question. In general, your assessment of MD5 manipulation is spot on. But I believe the value of MD5 checksums on downloads is superficial at best. Perhaps after you download a file you can check the MD5 you have against a website, but I tend to see side-by-side MD5 storage as a “receipt” or something that is nice to have but not reliable. As such, I have generally never cared about MD5 checksums from downloaded files, but I can speak from my experience creating ad-hoc server-based MD5 processes.

Basically what I have done when a client wants to sweep a file system for MD5 checksums is to have them be generated into CSV files that maps filename, path, MD5—and other sundry file info—into a structured data format that I then have ingested into a database for comparison and storage.

So using your example, while an MD5 checksum might sit next to a file in it’s own text file, the authority record MD5 checksum would be stored in a non-connected database system. So if someone somehow hacked into a fileshare to manipulate data, that intruder would not have any access to the MD5 authority records or the connected history.

Recently I discovered a nice piece of library archival software called ACE Audit manager which basically is a Java application designed to sit and watch a filesystem for changes. It logs changes via MD5 changes. And it operates on a similar philosophy as my ad-hoc process—store the checksums in a database—but it takes it a step further but creating an MD5 checksum of MD5 checksums which is known as a hash tree or Merkle tree.

So let’s say you have 5 files in a collection. Those 5 files in ACE Audit manager would then get another—let’s call it “parent”—checksum that is a hash generated from the 5 MD5 checksums of each file. So if someone were to tamper with just one file, the hash for the file would change and so would the hash for the whole “parent” collection.

In general the way you need to look at MD5 checksums and related integrity hashes is unless they are not connected to some non-direct storage for the MD5 hashes themselves, they can be corrupted. And their value as a long term data integrity tool is equivalent to a cheap lock that comes “free” on a new piece of luggage; if you are serious about locking your luggage you will get a lock that cannot be opened in 5 second with a paperclip.

Answer 11

speed

Often people find it is much faster to (a) download a large file from some untrusted "nearby" content delivery network (CDN), mirror site, torrent peers, etc. and also download the corresponding short checksum file (often SHA256; older software often used MD5) from a few trusted sources. They find it unbearably slow to (b) download the entire large file directly from a trusted source.

validation

Often that person finds that everything validates -- the sources (trusted and untrusted) agree on the same checksum, and running shasum (or md5sum) with any of those short checksum files (it doesn't matter which one, when they are all identical) indicates that the large file has a matching checksum.

modification

You are right that when Mallory maliciously alters a large file sitting on some download site, it would be easy for Mallory to also maliciously the checksum for that file on the same download site so that shasum (or md5sum) run on that malicious checksum file would seem to validate the large file. But that checksum file is not the (only) one the downloader should use for validation.

When the downloader compares that malicious checksum file to the checksum files downloaded from trusted sources, if the original checksum slips through even one time, then the downloader will see that everything does not validate, and will know that something has gone wrong.

As cpast has said before, if a good cryptographic checksum is transmitted over a trusted connection, it can provide security (which is derived from the trusted connection).

As supercat has said before, the checksum files one site don't help people who download large files from the same site and in the same way that they download the checksum files -- they help people who want to download files from some other site.

"In terms of security, cryptographic hashes such as MD5 allow for authentication of data obtained from insecure mirrors. The MD5 hash must be signed or come from a secure source (an HTTPS page) of an organization you trust." -- https://help.ubuntu.com/community/HowToMD5SUM

Cryptographic checksums are an important part of practical public-key signatures (as implemented in GnuPG and other OpenPGP-compliant software). Public-key signatures have some advantages over the checksums alone.

Answer 12

I have heard this [MD5 checksum] is to allow [...] also for any malicious changes to be easily detected.

Well, you haven't heard totally wrong. Digital signature is actually what's used to detect malicious changes. For some important reasons, hashing is a fundamental part of digital signature, in that only the hash is actually signed, not the entire original file.

That being said, if the source doesn't provide the signature of the hash and a trusted way to verify it, then you're right, no protection is given against deliberately altered files, but the hash is still useful as a checksum against accidental corruption.

Here's a real world example that may clarify the whole thing. The following passage is particularly significant on the subject:

For older archived CD releases, only MD5 checksums were generated [...] For newer releases, newer and cryptographically stronger checksum algorithms (SHA1, SHA256 and SHA512) are used