is there a filesystem that constantly looks for ways to compress data using a small amount of processing power?

particularly i'm looking for a way to dedupe files within various archives (.bz2, 7z, .zip) across many drives, or otherwise the compression method finding a more efficient way of compressing similar byte streams

i want to be able to copy a file to the new filesystem and then if it's a duplicate byte stream that the compression filesystem would eventually compress it, to the point that it may as well be a symlink

ideally, when first copying a file it wouldn't be compressed, or otherwise a very small amount of compression (level 1) and then gradually reprocessing the file up until maximum compression (level 9) while always having read/write access

filesystem needs to have read-write access at all times for all files

  • doesnt exist and never will – Keltari Oct 29 '19 at 3:36
  • like most of my programming projects, it will exist one day if i get funding, and i'll make it open-source – loud_flash Oct 29 '19 at 7:08
  • This could be accomplished without much effort. There just isnt a demand for it. – Keltari Oct 29 '19 at 7:10
  • you're right. with everything going to the cloud, it just isn't economically feasible to have such a filesystem for a nas, but i'll do it anyway. one day. – loud_flash Oct 29 '19 at 10:00
  • 1
    its has nothing to do with the cloud. Simply adding a new drive to a NAS, or upgrading to larger drives provides immediate storage. Compression and data deduplication work. However, their benefits cant be predicted. You could spend massive amounts of time compressing and recompressing, deduplicating, for little to no benefit. Why waste all that time and CPU cylces when you can just slap in some relatively cheap hardware with immediate and predectible results. – Keltari Oct 29 '19 at 10:11

@Daniel-k suggested windows server (2012+) data dedup is what i'm describing, and for all intents and purposes, it's similar to what i'm after; further information is available here: https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable

@keltari suggested zfs, the open-source file system, supporting both compression and data deduplication, and it's exactly what i'm after. i'm now going to experiment with freenas to see the savings on data i can achieve.

  • i should note zfs is not exactly the same as openzfs – loud_flash Nov 2 '19 at 2:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.