I use Ubuntu 16.04.4 LTS (converted to Server from Desktop) installed on a ZFS filesystem which itself is on an Samsung 840 Evo 120 GB SSD. I also have four HDDs which are two mirrors (1 TB and 2 TB mirrors) that are also ZFS.
I use the system for storage, but also virtualisation. I use KVM virtualisation on Ubuntu and for one of the VMs I store the VM OS disk image (which is mounted on '/') on the SSD and the storage disk image (which is mounted on '/home') on the 1TB HDD mirror.
The system has 16 GB ECC RAM, and on the SSD there is an 8GB SWAP volume (ZLE compression). The VM is set at 8192 MB, so with other services on the host it gives approx. 7.8 GB for ZFS ARC.
I have noticed that when writing a lot of data in the VMs to the VM disk images, the ZFS ARC grows massively in size. While this is expected behaviour of ARC since the data is getting hits, I do have some questions to ask regarding the use of ZFS in relation to storing and using VM images that I am not sure about below.
Please note the following:
- The VM disk images are not storing any important data - they are merely used for hosting and experimenting so the integrity is not so important;
- The disk images are stored in a filesystem within a filesystem - so if the directory is '/MyMirror/VM/', then 'MyMirror' is the initial filesystem and the latter ('VM') is the filesystem within;
- The disk images are of the RAW format (*.img).
Taking into account the above and considering the case ...
- Is there actually any benefit to having the VM disk images in ZFS compared to ext4?
- Is there any logical reason/benefit to the ARC caching the VM disk image?
- Am I doing it right?
Number 2 is baffling me the most at the moment as I am struggling to see the logic and any potential benefits of the ARC caching the disk images. I could be missing the point entirely (perhaps a cloudy mind), but how is caching the disk image on the host helping the host or even the VM itself?
I understand number 3 may be more of an opinion thing (which I wanted to avoid), but I am unsure if I am going about this the "right" way. Perhaps there is a better way, taking into account my setup, to store and use VMs.
Thanks in advance!