How to copy folders on same NTFS drive using hardlinks?

Question

Is there a way to create hardlinks to create a duplicate of a folder on the SAME drive (NTFS formatted)? I am looking to do something like this: XCOPY E:\TEMP\SAMPLE1* E:\TEMP\SAMPLE2\ /s but INSTEAD of actually copying the files and using up the extra drive space, just create hardlinks to the files.

Goal: there are only a handful of files that we do not need in the SAMPLE2 folder but want to keep all of them in SAMPLE1. The folder SAMPLE2 will eventually be copied off to another drive.

NOTE: the total content of SAMPLE1 is about 400GB (with multiple sub-folders contained therein) and each of the files that we are going to exclude from SAMPLE2 is about 20-40MB each (there are lots, 100+ files, located in various sub-folders).

So is there a way to do the equivalent of XCOPY (command above) except use hardlinks, and then we go and remove all the files we don't want in SAMPLE2 before copying them off to the external drive; then deleting SAMPLE2 folder.

Answer 1

This is simple in theory. You need to create a symbolic link of the directory(s) which will do EXACTLY what you want. It copies shortcuts that look and act like real files to the user and programs. Then, you can delete the symbolic links you do not want which only deletes the links; then you copy the rest of the "files" (really links) like normal and it will copy the actual files.

But windows is GUI based, and there is no built in GUI solution to make symbolic links. Also, the command prompt made it difficult as well. (If your using Linux, its easy and straight forward via the terminal, and some graphical file managers like GNOME Commander makes it easy through the GUI as well.)

Sooo.. to execute the above solution the "easy" way is to download a plugin and support file (if needed; read download notes) here This will add symbolic link options to your right click menu. The ones you want are "Pick Link Source" to 'copy' the directory, then "Drop As > Hardlink Clone" to paste. Work in the linked directory as you want deleting files (links) and such. Then regular copy and paste it (NOT pick link and drop as) to where ever you want and Voila, shortcuts copied as files!

Finally, delete the linked folder and it's like nothing happened.

The plugin seems to work great. Try it out! Just be careful to make Hard Links ONLY (No soft links or junctions) and be sure to work with the links and not the real files!

Now for those who would mention mklink, here is what I ran into while developing this answer. Using mklink via the command prompt running as an administrator (search for CMD and right click and run as administrator) SHOULD work the same... But it didn't for me (and MANY others if you search). I got an access denied even after modifying privileges via secpol.msc. It seems to be a prevalent problem.

Answer 2

Kind of found a "hack" method of getting the results we want for now.

There is a free utility called DirectorySlicer for Windows that helps us "splits" (really it makes "copies") of files into directories of a specified sized-chunks. A bonus feature of this utility is that it uses "hardlinks" if the destination drive is the same as the source and is NTFS.

So here is the "hack" we just did. We specified a new destination and specified a target directory size of 1TB, effectively larger than the entire source so it creates one directory. Seconds later we have a bunch of directories with hardlinks to the source files, just want we wanted (almost, it create an extra directory level, but that can be fixed easily enough).

Answer 3

It's a bit of a hack but:

1. Make a regular full copy with the GUI. Copy/paste etc. Watch your disk space disappear.
2. Run DupeGuru with the source set as reference and the dupe as "normal"
3. When the report is complete do "select all > delete" and tick the option "recreate as hard links". It can also do symlinks. You should recover some disk space. I use this sometimes to prepare a folder to upload after deleting a lot of other files in the structure with a script.

BONUS: It runs on Mac, Windows, and Linux. Nice! https://dupeguru.voltaicideas.net/

Answer 4

No 3rd-party software needed. This can be accomplished with PowerShell. Here's a pipeline-enabled function that takes a source path ($Path) and a "mirror" path ($MirrorPath). It populates the mirror path with hardlinks to files in the source path and subfolders to match the source subfolders, which are recursively populated as well.

Function Mirror-Path {
  Param(
    [Parameter(Mandatory,ValueFromPipeline,Position=0)]
    [String[]]
    $Path,
    [Parameter(Mandatory,Position=1)]
    [String]
    $MirrorPath
  )
  Begin{
    If ( ! ( Test-Path $MirrorPath )) {
        mkdir $MirrorPath -Force | out-null
    }
  }
  Process{ 
    ForEach ( $Target in $Path ) {
        If ( Test-Path $Target ) {
            Resolve-Path $Target | gci -File | ForEach {
                New-item -ItemType Hardlink -Path $MirrorPath -Name $_.Name -Target $_.FullName | out-null
            }
            Resolve-Path $Target | gci -Directory | ForEach {
                Mirror-Path $_.FullName ( Join-Path $MirrorPath $_.Name )
            }
        }
        Else {
            Echo "'$Target' is not a valid path."
        }
    }
  }
  End {}
}

Answer 5

This can be done with cp or rsync - e.g. from the widely used MSYS2 collection (cp is in the default installation. Add rsync: pacman -S rsync )

Simple file tree copy with files as hardlinks:

cp -al SRCDIR DSTDIR

This preserves timestamps etc. including NTFS permissions and alternate data streams - without cp knowing about those because they are stored in the MFT whose entries are shared for hardlinks.

Updating with rsync as hardlinks

rsync -a --link-dest=SRCDIR_RD SRCDIR/ DSTDIR

With absolute paths the SRCDIR_RD can be equal to SRCDIR.
With relative paths the SRCDIR_RD must be the path to SRCDIR relative from DSTDIR.
Absolute paths for rsync (unlike cp which understands normal drive letters) so far need to be given in old MSYS style ( e.g. /C/sub/dir or /C\sub\dir instead of C:\sub\dir)