14

I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.

I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.

In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.

To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.

Is there something like this?

9
  • 1
    It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware. Oct 21, 2012 at 7:47
  • 1
    If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way! Oct 21, 2012 at 8:03
  • 3
    I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
    – Alfatau
    Oct 21, 2012 at 8:06
  • 2
    Yes, in fact in my experience is happens really more frequently then a sudden drive failure. Also, i had many disks with damaged clusters (and lost files) while only one has suddenly failed.
    – Alfatau
    Oct 21, 2012 at 8:09
  • 1
    Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
    – Alfatau
    Oct 21, 2012 at 8:13

4 Answers 4

4

I'm not sure about virtual filesystem but there are many "real" file systems that have built-in fault-tolerance

  • Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
  • Btrfs – A file system based on B-Trees, created by Oracle Corporation.
  • HAMMER and HAMMER2 – Dragonfly BSD's primary filesystem, created by Matt Dillon.
  • ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
  • Reliance – A transactional file system with CRCs, created by Datalight.
  • Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
  • NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
  • ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
  • WekaFS – a shared parallel filesystem that delivers extreme performance at any scale and is optimized for NVMe and the hybrid cloud.

File systems with built-in fault-tolerance

I don't know the detail features of those files systems so some of them may only provide data error detection instead of correction. You need to check their documents. But if they do have error correction then of course you can use them in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.

For Btrfs you can also have data duplication for error correction although it'll cost twice the disk space. Just use --dup data option

mkfs.btrfs --metadata dup --data dup /dev/<device>

In ZFS it can be done similarly with the set copies command

zfs set copies=2 users/home

For more information read How to get error detection and correction on a single hard drive on linux (with btrfs or other methods)

1
  • 1
    error fault is good but the question was about the error correction (detect+correction) like using ECC or with Reed Solomon algorithm. Duplicating data is less efficient than having ECC/RS algo based filesystems that require a percentage of the size not twice or triple of the space.
    – None
    Nov 14, 2021 at 5:36
1

Yes; Ceph

is the only "file system" I'm aware of that optionaly uses Error Correction Code (ECC). Normaly ECC is implemented in hardware (RAM, HDD, SSD) so btrfs/zfs/stratis only use checksums, mirrors, parity to handle full drive falure, or corruption past the ECC threshhold. Unlike btrfs/zfs/stratis, Ceph and other distributed file systems have no inherent single point of falure so Ceph/etc can offer continuous availability.

Some archives (rar and par2) offer ECC. like .md5 files can accompany other files so can par2.

I recommend reading https://en.wikipedia.org/wiki/Error_correction_code

1

See https://www.thanassis.space/rsbep.html

This creates extra files with data for error correction using the Reed-Solomon error correction used on audio CD's in the past.

Too pity there's no filesystem which uses this technique.

1
  • no filesystem which uses this technique but there's a Reed-Solomon FS (a FUSE-based filesystem) section in your link
    – phuclv
    Apr 13, 2021 at 15:50
0

I haven't heard of one that works like a block-level filesystem.

Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.

You could always make your files redundant yourself using a PAR2 tool such as QuickPar.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .