2

Is there a redundant RAID/ZFS-like filesystem that can optimize the placement of files based on their access patterns? My goal is to build a desktop system that will automatically move files among a hard disk, a solid state drive, and a RAM disk based on how and how often they're accessed, in order to control wear through opportunistic volume selection. For example, files that are read frequently (and opened for read-only access) would be placed on a SSD; files that are read frequently, but are opened for read-write access, would be placed on a RAM disk; files that infrequently accessed would move to the HD.

I'm imagining a smart virtual filesystem that can automatically recognize access patterns and optimize physical volume placement. However, I get the sneaky feeling that a similar effect could be accomplished by creating a primary filesystem on the hard disk, a swap partition on the SSD, and reserving a large quantity of RAM for a filesystem cache.

The downside to the above (easy) solution is that that it's insensitive to the weaknesses of the underlying hardware. For example, writes cause SSDs to fail more quickly - and HDs are even more susceptible. RAM, however, can be written almost indefinitely without failing. The filesystem should capitalize on these properties to maximize the lifetime of the equipment.

3 Answers 3

3

You should take a look at Btrfs, especially the balancing portions. It may include code that's either exactly, or close to, what you need, and that you can build on.

The system as you describe it is almost cache-like, and can certainly be enforced as an add-on to existing file systems to some extent. Some things of note, in general:

  • Disk controllers don't propagate writes immediately, which means writing the same file rapidly in a short amount of time won't actually cause the underlying hardware to be updated as many times.
  • SSDs aren't as feeble as they once were, and can survive as much beating as a HDD, if not more. The lack of moving parts is great!
  • Your proposed design is only useful if access patterns don't change too often, just like any other cache. Async I/O has sped up perceived disk interaction times in recent years, but it depends on application-level support sometimes. On the other hand monkey-patching the OS to serve the in-RAM copy of the file to applications, while doing AIO to the HDD, might give the kind of data durability that a purely in-memory system won't give you (see: memcached, and why it strongly recommends against being used as a persistence engine)

I understand that the end-goal is to be as transparent as possible to the OS/user/applications, to present just one volume/drive/partition and take care of everything internally. I'm not sure if the desktop is the best target for such a system, because I doubt it would be as useful as if it was built for servers.

P.S. This has been around in hardware as hybrid drives for some time now. The RAM part isn't used, but then again desktop users wouldn't really notice it anyway. Another proposal is the Combo Drive and Google Scholar has yet others.

1
  • Hybrid drives seem like a slick idea, but I don't think the performance economics are really there. I agree that this kind of multi-stage caching is useful in server environments but not for most desktop users. Apr 29, 2011 at 5:25
3

ZFS is already doing this kind of optimization automatically. Just add a SSD as a cache device in your pool and your files (actually data blocks) will be located, from the less to the most active ones:

  • on the regular disks
  • on the SSDs (L2ARC)
  • on RAM (ARC)

The algorithm used by ZFS ARC (Adaptive replacement cache) is based on both the most frequently used blocks and the most recently used ones.

http://blogs.oracle.com/brendan/entry/test

1

You might want to look into bcache: http://bcache.evilpiepirate.org/

It looks promising but I didn't try yet.

2
  • very interesting! i haven't finished reading, yet, but i wonder if it can also use RAM cache ahead of the SSD. that would be perfect! Nov 6, 2011 at 18:18
  • I don't think that putting bcache on a RAM disk gives any benefits - this is where the linux file system cache excels. Bcache goes much deeper in allowing having a huge cache (compared to RAM) optimized for random access which SSDs excel at (compared to hard disks). Bcache expects to work on non-volatile storage - RAM disks are thus not your best choice although it would be possible.
    – hurikhan77
    Nov 7, 2011 at 16:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .