The question may be imprecise so I will try to explain it in more detail.

For a number of reasons I have lots of copies of the same file on my Linux file-system. Many of them are quite large.

Say I have /path/to/some.file and copies of this file /other/path/file.name and /yet/another/path/third.copy. I wonder if there is a file-system which would literally make two of these files act as a reference to the original. Naturally, if user modifies one of them, then and only then they become independent files.

PS. I know this can be (partially) accomplished by using links. But I want this feature I tried to explain above to be transparently handled by the file-system.

  • Since there doesn't seem to be a perfect answer yet, why not write a script (perhaps run from cron) that replaces files with hard links (or, if it's safe for some of them, symlinks)? In any case, if you don't already know where all the duplicates are, check out fdupes code.google.com/p/fdupes. – Joe Nov 19 '12 at 19:09
  • 1
    I can't do that because when some user changes data in one of duplicates, they should become independent files because the content is different after the change... If i make symbolic links, then modification will actually change the original. Think of the original file as a value in a functional programming language - when you modify it, you get literally a new object, and reference to it. – DejanLekic Nov 22 '12 at 15:45
  • OK. Just some ideas: Is there some way you can get between your users and these files? If you can, you could start with a link (sym or hard) and then break the link if they write anything to the file. You could also use something like diff to just store differences to save a lot of space like in a version control system. If you put the files in one of those, it could manage the differences for you. I haven't used vcs systems much, so I don't know the details. – Joe Nov 23 '12 at 19:42

This feature is called deduplication. None of the popular Linux filesystems (ext*) support it, but apparently, ZFS supports it partially. There is also a table of filesystems listing, among others, deduplication, but there don’t appear to be any popular choices - it is a planned feature for Btrfs, though.

I would guess that periodically checking your filesystem and creating appropriate hard links is the best you can do at the moment, although that does not imply copy-on-write.

| improve this answer | |
  • I hoped for BTRFS to have it alraedy. I did not know they plan to have this feature - good news! Thanks! – DejanLekic Nov 16 '12 at 13:56

The primary keyword you want to look for is "copy on write." BTRFS does have a clone operation that does exactly what you want, and cp --reflink will do what you're looking for, provided your system has a modern enough kernel and coreutils 7.5. Wiki Source Also, bedup is a tool that will merge duplicates over an entire volume. CoW is also the driving feature underneath btrfs's snapshotting technology, IIRC.

| improve this answer | |
  • I am aware of cp --reflink but note that I actually want the FS to detect clones and use references instead, transparently. I am also awayre of the bedup tool. – DejanLekic Nov 16 '12 at 16:02
  • When a user-level process reads from one file and writes to another, how is the FS layer going to know that it's an exact copy and not a modified copy? That's why there's a separate ioctl for cloning vs. just reading and writing. Applications that want to clone files and want to use CoW should use such things when available and fall back otherwise. There's no reliable magic method by which the FS layer can detect a copy happening and do CoW instead. An app must either use the proper call for the job (cp --reflink/clone) or a separate tool to finish the job later (bedup). – afrazier Nov 16 '12 at 16:24
  • By comparing hashes (more safe) or crc+timestamp values. Write would definitely cause FS to create an independent copy of a file. Futhermore, this system ma be implemented on the page level... – DejanLekic Nov 16 '12 at 16:51
  • And where exactly in the kernel is that done? What you're talking about isn't a simple problem. Doing it at the page level is also possible, but requires massive amounts of RAM to do inline -- ZFS may be more than 20 GB of RAM per TB of data [Source], and OpenDedup wants 8 GB of RAM per TB of data w/ 4k pages [Source]. ZFS's requirements can be reduced with L2ARC at the cost of performance. – afrazier Nov 16 '12 at 18:13
  • 1
    WHS doesn't do dedup, unless you're talking about the backup store. In the case of backup software, the client can do most of the work and check the hash with the server before sending any data to it to be written. A backup service is a very different beast than a filesystem. Strategies that work for one may not be viable for another. – afrazier Nov 16 '12 at 19:31

There is an online file system S3QL designed for backups with great capacity of deduplication.

| improve this answer | |
  • I should have said that I need this for a local filesystem... +1 anyway for useful information. – DejanLekic Nov 22 '12 at 15:43
  • @DejanLekic, urls like local://... are allowed. – Daniel Fanjul Nov 23 '12 at 19:16

Zfs, btrfs, ext3cow, bcachefs (afaik, but there is a chance it's not yet implemented). Microsoft had one in development but they stopped for unknown reasons.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.