How to copy files and create hardlinks instead of copying when files are identical

Question

Are there any tools for Windows that can

• copy one dir to another
• read copied content
• generate MD5
• if the current file is identical to a previously copied one, create a hardlink in destination dir instead of writing the content?

Answer 1

If you're copying the content just to hardlink it immediately afterwards, why not just generate the hardlinks straight away? Link Shell Extension makes this particular job easy.

If there's a reason you need to go through that particular sequence of actions, LSE's author also wrote a command line tool called dupemerge to do almost exactly what you're asking.

One thing to keep in mind is that NTFS does not do "copy-on-write" semantics for hardlinks. If something modifies the contents of a file, all hardlinked versions are immediately "updated", since they're all essentially directory entries to the same data extent on disk. What's more, many programs do a "save to temp file, delete original, rename temp to old name" procedure rather than overwriting a file, which will effectively break other hardlinks to the data, since they're pointing at the old data extent.

Answer 2

You can do this using FINDDUPE, which you can find here.
Consider src as your source folder and dest as your destination folder you can do:

xcopy /I /E src dest
finddupe -hardlink -ref src dest

Note: Hardlinks only work on NTFS