30

I have a directory with a lot of books in PDF format (around 2GB in size).

While reading I often leave notes and annotations in the files. Occasionally I make a backup on an external hard drive. Since I do not remember which files have been modified since the last backup, I just copy and overwrite the whole thing.

Does Windows check if files with the same name are identical (by content) before overwriting? If not, how would I approach doing exactly this?

0

6 Answers 6

63

Robocopy.

Windows cannot differentiate between identical and modified files if you copy using Windows Explorer.

Windows can differentiate between identical and modified files if you copy using Robocopy, which is a file-copy utility included in Windows (Vista, 7, 8.1 and 10).

There's no need to use third-party tools.

You can save this script as a batch file and re-run it whenever you want to perform a backup:

robocopy /e c:\PDFs p:\PDFs

  • Whenever a PDF file is annotated and the changes are saved, both its Last Modified and Size attributes will change in the source folder, triggering an overwrite of the corresponding file in the destination folder the next time Robocopy is invoked.
  • Robocopy compares the Size and Last Modified attributes of all files that exist in both the source and destination. If either attribute is different, then the destination file will be overwritten. Note: these values only need to be different, not necessarily newer or larger; even if a source file has an older Last Modified or smaller Size attribute, the source will still overwrite the destination.
  • the /e (or /s) switch isn't necessary if all of your PDFs are in the root of the folder referenced in the script but if you want to include Subfolders then use /s. If you want to include subfolders and Empty subfolders, then use /e.
  • I would assign the backup drive a letter further along in the alphabet, so there's no risk of another drive being inadvertently assigned the drive letter used in the script, causing the backup to fail at some point in the future. I used P here for PDF.

That simple script is all you need.

8
  • 9
    Is it really wise to "skip" failed copies when making backups? Oct 14, 2019 at 14:19
  • 7
    Just a side note: Rococopy does not check the content, it only checks metadata. But yeah, in this case I suppose robocopy should suffice.
    – Albin
    Oct 14, 2019 at 16:10
  • 4
    @Jacob Raihle good question. The retry and wait parameters have default values of 1 million attempts and 30 seconds respectively. You can always use smaller values for both but locked files often require user intervention to unlock and reattempts in the short term generally fail regardless. So this is personal preference but I like to let RC race through all the files without getting hung up for a few minutes on a single one. If a file does fail, this will be reported in the job summary report anyway. In general, I find that tomorrow's backup will scoop up any file that was locked today. Oct 14, 2019 at 19:21
  • 1
    overwrite every PDF on your backup drive that doesn't have a matching Last Modified timestamp does that take into account if the destination timestamp is newer than source? It doesn't seem relevant in this question, but still worth knowing.
    – Gary
    Oct 15, 2019 at 15:11
  • @Gary Robocopy doesn't distinguish between older vs. newer or smaller vs. bigger files, it only cares that they are different. So even if a file in the destination folder is newer and/or bigger, Robocopy will still happily overwrite it with the older and/or smaller file from the source folder. It senses that the files are different and resolves the difference by pulling data across in the same direction every time: source to destination. Oct 16, 2019 at 0:04
6

Windows does not do this. It will however prompt you to overwrite files with the same name and you can select manually if you want to do it.

For a easier solution, use FreeFilesSync to compare the folders and overwrite only changed/updated files (Mirror option and select File time and size in Comparison Settings).

4

Yes and no! Windows Explorer only checks for metadata (file size, dates etc.).

But you could use a script e.g. powershell (see here) which comes with (most) Windows or 3rd party tools that let you compare/copy files using file checksums e.g. MD5 or SH1 hashing (see here and/or use a search engine).

I myself like to use the software checksum compare (see here), it lets you compare files and directories including file checksums and it works from a USB pen drive.

If you don't need to compare the file's content and if you just want to copy "newer" files you can use any advance copy method like xcopy, robocopy, etc.

Note: the different hashing methods have up and downsides (mainly reliability vs. speed). For me MD5 is more than enough for this type of file comparison, but that's a personal preference. See here for further info on that topic.

0
3

In short: No

Windows doesn't dot that in a straightforward way.

Well, it does, but like everything in Windows it's ambiguous at best. You will be prompted for name conflicts, and depending on your Windows version, you get a more or less understandable dialog with several options to choose from, with an additional note ("Blah blah, different size, newer"). You can then, one by one, choose whether or not to keep the modified file, and you have the option of applying this to all "identical" matches.
Now of course it's Windows, so you have no guarantee that "newer" actually means newer, and you do not know what is "identical" (is it just the name collision, is it the size change, is it the modification date, or is it everything?).

Alternatives

There exist a huge variety of file sync programs, both free and commercial which are somewhat better insofar as they check whether a file has been modified before overwriting it, rsync being the traditional mother-of-all-tools, but also being a tidbid less user-friendly than some people may wish.
However, I do not recommend any of these because they are not fundamentally making things better.

Personally, if you are not afraid of a little commandline (could always make a batch file!) I'd recommend Matt Mahoney's excellent zpaq. This is basically ZIP, except it compresses much better, and it does deduplication on the fly.

How it that better?

Well, checksum-comparing tools are all nice and that. Especially when you go over the network, nothing can beat rsync running on both ends, it's just awesome. But while a typical sync tool will do the job just fine (and better than Explorer) this is not what it's best at.

Writing to an external drive, whether or not you compare checksums, has a couple of things you need to cope with:

  • Access time on the drive (abysmal)
  • Latency over USB or what you use (getting better but still kinda abysmal)
  • Bandwidth (actually pretty good nowadays)
  • Drive writes (and amplifications)
  • Drive reads

In order to compare checksums, you first have to read in the files. Fullstop. Which means that for a couple of thousand files, you pay for the latency of traversing the directory structure, opening files over a high-latency link, and reading the files several thousand times. Plus, transferring them in small units over a high-latency wire. Well, that sucks big time, it is a very expensive process.

Then, you must write the files that have changed, again with several high-latency operations such as opening files, and overwriting data, and again one by one. This sucks twice because not only is it inherently unsafe (you lose the file being overwritten if your cat stumbles over the USB cable) but also with modern shingling harddrives (such as many external drives), it can be excruciatingly slow, down to single-megabyte-per-second if you are unlucky. That, and the latency of thousands of small transfers adding up.
A well-written file copy tool may be able to deal with the safety issue by copying a temporary file, and atomically renaming it afterwards (but this adds even more overhead!).

Now, an archive format like zpaq will create an archive that contains the checksums of the files already, they can be read in quickly and sequentially from one location. It then locally (locally means "on your side of the cable" where you presumably have a reasonably fast disk connected via SATA or M.2 or something) compares checksums, compresses differences, and append-only writes the compressed data sequentially to the existing archive. Yes, this means that the archive will grow a little over time because you carry a whole history around. Alas, get over it, the cost is very moderate thanks to diffing and compression.

This method is faster, and safer at the same time. If you pull the cable mid-operation, your current backup is interrupted (obviously!). But you do not lose your previous data. All transactions that go over "slow" links are strictly sequential, large transfers, which maximizes throughput.

0
1

XCOPY/D will only copy files if the source is newer than the destination. (XCOPY/S/D for a recursive copy)

1
  • XCOPY does not take the contents of the files into consideration
    – Bert
    Oct 16, 2019 at 13:24
0

Microsoft SyncToy

Microsoft makes a great Windows PC app called SyncToy where you specify pairs of "left folder and right folder", and choose to echo changes from left to right, contribute changes from left to right without deleting right, or synchronize between left and right. There is a user interface for previewing the changes before committing.

If a file is detected as identical, it will be skipped over, which is the functionality that you are looking for.

I have been using the Echo mode for about 10 years to incrementally mirror changes from my desktop PC to an external drive.

https://www.microsoft.com/en-us/download/details.aspx?id=15155

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .