Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Are there any tools for Windows that can

  • copy one dir to another
  • read copied content
  • generate MD5
  • if the current file is identical to a previously copied one, create a hardlink in destination dir instead of writing the content?
share|improve this question
up vote 3 down vote accepted

If you're copying the content just to hardlink it immediately afterwards, why not just generate the hardlinks straight away? Link Shell Extension makes this particular job easy.

If there's a reason you need to go through that particular sequence of actions, LSE's author also wrote a command line tool called dupemerge to do almost exactly what you're asking.

One thing to keep in mind is that NTFS does not do "copy-on-write" semantics for hardlinks. If something modifies the contents of a file, all hardlinked versions are immediately "updated", since they're all essentially directory entries to the same data extent on disk. What's more, many programs do a "save to temp file, delete original, rename temp to old name" procedure rather than overwriting a file, which will effectively break other hardlinks to the data, since they're pointing at the old data extent.

share|improve this answer
> why not just generate the hardlinks straight away? because it's two HDD, one removable, the other one have limited free space left – est Oct 17 '11 at 6:25
You should edit your question with your restrictions and the particular sequence of actions you'd like to take. Are you sure there's enough duplicate files to allow for what you want to do? If so, you could use dupemerge on the source, then LSE or it's CLI companion ln to do a Smart Copy that will preserve the interior links. – afrazier Oct 17 '11 at 12:57
du on the source is good idea. thanks :D – est Oct 18 '11 at 5:25

You can do this using FINDDUPE, which you can find here.
Consider src as your source folder and dest as your destination folder you can do:

xcopy /I /E src dest
finddupe -hardlink -ref src dest

Note: Hardlinks only work on NTFS

share|improve this answer
thanks, I know finddupe, but the HDD target has limited free space and source HDD have lots of dup files. How to check BEFORE copy? – est Oct 17 '11 at 6:26
I don't think you can make hardlinks between two different HDDs. Is that what you want to do? – qwertzguy Oct 17 '11 at 10:04
copy files to another HDD, if dup, create a hardlink instead of copy content. – est Oct 18 '11 at 5:24
Ah ok... now I get it: you want to check duplicate files, then copy only non duplicates and make hardlinks to recreate the reste of the files. Well you can use finddupe -bat find.bat src then in the created file find.bat remove all the del lines, make a unique list of hardlinked files (last parameter of fsutil) and copy those files, then run the script. Of course a script to do this would be best, but I don't have time to do it right now. – qwertzguy Oct 18 '11 at 9:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .