Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Git stores content uniquely in its repo based on the calculated hash of any file. Thus, if my directory has two copies of the same file somewhere inside it, git will only actually store it once.

I am wondering if this same concept has been implemented at the operating-system level as some kind of file system?

One of the sweet spots in Unix systems are symlinks because they effectively allow content to exist in multiple places at the same time without the added byte cost of actually storing the same content multiple times. If a file system was smart enough to do this automatically, this would seem to pose a good possible solution for dll hell issues.

Ruby enthusiasts share libraries by publishing them as rubygems. Still, this effort to share gems resulted in deployment nightmares that lead to the Vendor Everything concept of copying all dependencies into local folders to avoid such nightmares. I am wondering secondly if such a file system does exist, wouldn't it be a good complement to the "Vendor Everything" approach? (This question is a tangent to the first and primary question.)

share|improve this question
2  
I'm not an expert, but check out ZFS. –  ForeverWintr Aug 9 '13 at 16:31

2 Answers 2

up vote 16 down vote accepted

What you're looking for is called "deduplication". While it's usually implemented by vendors of specialized storage products, the ZFS filesystem implements it as well. Most Unix-derived operating systems can make use of ZFS, and I'd therefore recommend it as the first place to look.

share|improve this answer
1  
I see "deduplication" can be implemented at the file-level which is what I was concerned with in particular. –  Mario Aug 9 '13 at 16:45

Network Appliance, Inc. has had storage ability like this for many years, in fact they had complaints filed with Sun Microsystems for their ZFS file system, and does what Aaron Miller is mentioning in his accepted answer. For what it's worth, the complaints were settled with Oracle after about 3 years.

I have used this as a corporate solution since 2000, and it works well. The cost of storage after the first 'copy' comes about once there is a change in the file. Otherwise, with many 'copies' comes only a slight increase for namespace.

I don't think this answers the question about "operating system level" but rather on a "file system level".

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.