11

I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.

I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.

In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.

To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.

Is there something like this?

9
  • 1
    It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware. – David Schwartz Oct 21 '12 at 7:47
  • Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost. – Alfatau Oct 21 '12 at 8:02
  • 1
    If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way! – David Schwartz Oct 21 '12 at 8:03
  • 2
    I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data. – Alfatau Oct 21 '12 at 8:06
  • 1
    Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support. – Alfatau Oct 21 '12 at 8:13
4

I'm not sure about virtual filesystem but there are many "real" file systems that have built-in fault-tolerance

  • Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
  • Btrfs – A file system based on B-Trees, created by Oracle Corporation.
  • HAMMER and HAMMER2 – Dragonfly BSD's primary filesystem, created by Matt Dillon.
  • ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
  • Reliance – A transactional file system with CRCs, created by Datalight.
  • Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
  • NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
  • ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
  • WekaFS – a shared parallel filesystem that delivers extreme performance at any scale and is optimized for NVMe and the hybrid cloud.

File systems with built-in fault-tolerance

I don't know the detail features of those files systems so some of them may only provide data error detection instead of correction. You need to check their documents. But if they do have error correction then of course you can use them in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.

For Btrfs you can also have data duplication for error correction although it'll cost twice the disk space. Just use --dup data option

mkfs.btrfs --metadata dup --data dup /dev/<device>

In ZFS it can be done similarly with the set copies command

zfs set copies=2 users/home

For more information read How to get error detection and correction on a single hard drive on linux (with btrfs or other methods)

1

See https://www.thanassis.space/rsbep.html

This creates extra files with data for error correction using the Reed-Solomon error correction used on audio CD's in the past.

Too pity there's no filesystem which uses this technique.

1
  • no filesystem which uses this technique but there's a Reed-Solomon FS (a FUSE-based filesystem) section in your link – phuclv Apr 13 at 15:50
0

I haven't heard of one that works like a block-level filesystem.

Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.

You could always make your files redundant yourself using a PAR2 tool such as QuickPar.

0

Yes; Ceph

is the only "file system" I'm aware of that optionaly uses Error Correction Code (ECC). Normaly ECC is implemented in hardware (RAM, HDD, SSD) so btrfs/zfs/stratis only use checksums, mirrors, parity to handle full drive falure, or corruption past the ECC threshhold. Unlike btrfs/zfs/stratis, Ceph and other distributed file systems have no inherent single point of falure so Ceph/etc can offer continuous availability.

Some archives (rar and par2) offer ECC. like .md5 files can accompany other files so can par2.

I recommend reading https://en.wikipedia.org/wiki/Error_correction_code

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.