As a backup procedure, I would like to periodically sync files (for instance via rsync) from a Windows machine to an external HD.

However, I was wondering about file corruption. If a file gets corrupted on the host, its checksum changes, rsync will see it as modified and will copy the corrupted file from the host, overwriting the good copy on the HD.

Does rsync have any kind of mechanism that prevents this from occurring? If not, how can this be avoided?

EDIT: To clarify, I need to tell whether the checksum changed because I added new content to the file or because the file got corrupted. In both cases rsync is going to see the file as "modified" and will write it over the old version on the HD. In the first case rsync is doing what I wanted, in the second it is erasing my old valid file with garbage.


The way you generally deal with this problem is the way you deal with any other situation in which you want to roll back to a previous version of a file: Incremental backups.

The idea here is that you only actually copy the file when the contents change. For any other copy of the file on the backup media, you just refer to a previous revision. This allows you to go back to a previous revision of the file if the current one turns out to be corrupted, the idea of course being that if you don't notice the corruption in time then the file probably wasn't very important to begin with.

rsync supports this (at least on *nix) with the --link-dest command-line parameter.

For this to work nicely, you also need something to clean up old copies, which rsync doesn't; you'd need something else for that. Otherwise your disk will eventually fill up and you will have to delete files from the target media manually.

And to answer the direct question: rsync has no real handling of corrupted files. It probably will handle files that cannot be read, unless you are running it in in-place replace mode, but it isn't going to look at that JPEG image or PDF document and determine that it is malformed causing it to look weird when rendered. That's just not its job.

Compare this note in an answer to Why is RAID not a backup? on Server Fault, by Jared Oberhaus:

Will a backup refuse to copy a corrupt file?

Even if a backup copies corrupt or bad data, the point of a backup is that you can and should have multiple copies. For instance, last hour, yesterday, last week, etc. You can get a similar effect from using rotating snapshots on your storage device.

and related, in the words of Jörg W Mittag in another answer to the same question, highlighting the difference between redundancy and backups:

If you accidentally overwrite your PhD thesis with garbage, redundancy ensures that you have multiple copies of garbage, in case one gets bad. A backup ensures that you can restore your PhD thesis.

  • The problem here is that if a corrupted file is saved on disk, all subsequent deltas will be worthless. – dr_ Mar 13 '17 at 9:06
  • @dr01 Yes. How do you expect rsync to know the difference between a file being intentionally overwritten with garbage and unintentionally overwritten with garbage? It's part of the answer to why RAID is not a backup all over again, but from a different perspective. When you notice the corruption, though, you'll still be able to reach into your backup and retrieve the previous version of the file, which will be unaffected by the corruption (except in the case where the corruption also affects the backup, but in that case you might have bigger problems..). – user Mar 13 '17 at 21:00

If you want to check about file content, you could use "--checksum" option. This option calculates a hash for every transfered file.

  • How do I tell the difference between a changed checksum because I willingly modified the file, and a changed checksum because the file got corrupted somehow? – dr_ Mar 13 '17 at 9:08
  • If you use rsync without --checksum option, and everything seems to be equal (this includes modification time, size, permission, etc), but --checksum shows difference, you could safely assume that the file suffered some kind of corruption. If file was legitimally modified, then some of those properties changes. – Roberto Paz Mar 21 '17 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.