Reasoning behind hosting virtual disk images on BTRFS Filesystem

Question

I have been reading about BTRFS and ZFS for a while and while I do appreciate the benefits of CoW when hosting important files I am currently puzzled by the following dilemma:

Given that both ZFS and BTRFS seem to suffer performance degradation when hosting large files subject to random writes (e.g. QCOW2, VDIs, etc), why would someone use BTRFS as a FS of choice to host VMs?

I know that in the case o BTRFS you can selectively turn off CoW using chattr, or the nodatacow mount option, reducing the degradation imposed by fragmentation and CoW...

However, according to Redhat this also "disables checksum verification of data and metadata, resulting in reduced data integrity." (in addition to the lost compression and naturally the so desired CoW).

Based on that I ask: Why would someone chose BTRFS over lets say, XFS over LVM over MD RAID ?

Answer 1

If the main use case is storing VM images or databases, and you are not interested in accepting the potential performance issues in order to get the data integrity advantages of btrfs, then I can't think of any reason why you would want to choose btrfs over xfs or ext4.

Disabling copy-on-write for just the VM image directory (using chattr +C) is mostly relevant when storing VM images is just one of many uses you have for your file system. Then it can be very convenient to just disable copy-on-write for that single directory, yet still retaining all of the advantages of btrfs for the rest of the file system.

Answer 2

BTRFS has a different on disk format that will outperform other file systems on some write patterns. In particular a large amount of effort has been spent in improving how meta-data is written to disk and it supports some advanced features such as data checksumming, compression, and snapshots. For large files improving meta-data performance is generally not important.

Compared to ZFS, BTRFS is a simpler solution and better supported on Linux. The major downside is that it does not scale as well (when adding large number of disks) and does not have the same feature set.

Compared to XFS it will be a lower performance solution. I.e. it will take more processor time to write a chunk of data to disk and it will be limited in maximum throughput. This can be mitigated to some extent by doing things like disabling checksum verification, but then you lose the primary benefit of BTRFS over XFS which is improved meta-data information. I.e. checksumming and different journaling (which may be better in some situations).

In terms of Copy On Write (COW) support, XFS favors performance over being strictly COW. I.e. XFS has very good meta data and journaling features in terms of scalability and will generally not overwrite file data on write with the exception being that it will allow existing disk blocks to be over-written if the application specifically requests to over-write that data. This can be good in the case of a VM as your disk allocation initially can be contiguous and in that case would remain contiguous for the lifetime of the VM.

Answer 3

Using BTRFS, even with nodatacow and alike you are still able to create snapshots of data manually with CoW behaviour, so it's fast and doesn't consume much memory. Additionally, those snapshots are more flexible than using LVM, because you don't need to reserve space underneath the file system which the file system is not aware of and can't be used if not needed at all. Like with ZFS, snapshots can be send and received over the network, which might help you improve your backup strategy. So even with nodatacow BTRFS seems to be superior over using LVM.

Answer 4

After having worked with cloud computing platforms I've come to adopt their view of virtual machines as a container for data and functionality where the actual vm is of less importance.

I'd rather have good backup-restore and install-deploy processes in place and worry less about the integrity of the VM and more about performance.

I.e. go for JFS/XFS/EXT4 over LVM over MD-raid rather than BTRFS. Make sure I never store anything that isn't backed up on the VM for any longer period of time... and have scripts and procedures in place to get a new install of the VM back up within reasonable time should something happen.

But this is more of a dev/experiment scenario than anything else.