Running a linux fileserver *as* a VM versus *in* a VM [closed]

Question

This is ultimately a question about how to optimize a physical->virtual machine move. I've found a number of posts that are somewhat related, but nothing (yet?) that appears to address it directly. Feel free to point and blame if I didn't search well enough... ;-)

My physical hardware consists of a half-dozen 2TB storage drives, an 8-core/32GB supermicro Xeon board, and a couple of reasonably fast SATA SSDs.

At the moment I have one physical machine that acts as a dedicated file server, running Ubuntu and ext4 over MDADM in raid5 mode. This is really the only "important" machine in my setup, as it's my local home backup and also serves as sort of a compute server for transcoding videos and other stuff like that. I also have a couple other small linux machines that I use for random, mostly unimportant stuff - I rebuild these often and if they break I don't really care as the important data is on the file server. I also have two older windows machines that I'd love to virtualize partially to save space/power but as much to get them access to much higher performance hardware.

I'd like to move the file server over to a more recent linux distro and also move the bulk of the storage over to a ZFS based filesystem. I'm familiar with ESXi from a work environment, but there I don't manage the storage so that part is a little opaque to me. I'm trying to figure out how to handle the storage on this setup. I can see at least these options:

• run linux on the baremetal. Set up that machine as the fileserver, with ZFS set up across the physical disks just as it would be on any other machine. Then run KVM on that machine, and host VMs from there.

• run linux on the baremetal, run KVM there and use this instance as basically just the hypervisor. From there build a VM that runs the fileserver and other VMs for all the other machines. In this case, I need some help figuring out how I'd expose the storage to the ZFS/file server VM. Do I just pass through all of these disks to the ZFS server? This would preclude any other VM from using this space, or they'd have to bounce back and forth a couple of times if they were hosted elsewhere.

• run ESXi on the baremetal, and launch one VM to be the file server and others for the other VMs. Here, mostly the same questions apply about how I'd go about exposing the storage to the file server. Is the performance likely to be better on ESXi vs. KVM? What about expanding the storage at some future time? I'm sure there's other stuff I'm not considering... but what is that other stuff?

Answer 1

ESXi can use local storage, an NFS mount, or an iSCSI LUN. If you install ESXi on the bare metal, it can access the local storage, where you would simply create one or more datastores (which you should be familiar with if you use ESX at work). ESXi does have one serious caveat, which is that it's quite picky about the network cards it supports. If you don't have an Intel or Broadcom NIC, it's not likely to work (as in, ESXi won't even recognize it). So check the hardware compatibility before proceeding in that direction. You also should check to see if your RAID adapter is supported. If either the NIC or the RAID adapter is not supported, that would be a showstopper as far as the ESXi option.

I am not familiar with KVM, so I can't comment on ESXi vs KVM or any KVM configuration steps.

Answer 2

You have pretty much figured it out already. All three alternatives are viable.

• Options 2 and 3 are essentially the same with different flavors, as ESXi is just a Linux distribution with virtualization software (roughly equal to another Linux distribution with KVM). The main difference between them is administration/maintenance (for example, ESXi has tools like vSphere to manage the host, KVM has different tools) and support for certain features (for example, certain Nvidia cards are certified to work as virtual GPUs while nearly all AMD cards work with ESXi, whereas KVM might have broader support, but might be more difficult to get it running, judging from tutorials on the Web).

• The difference between 1 and 2/3 is that the first is less complicated (in regards to how many layers you have), but more integrated (meaning you cannot easily replace the storage with something else, not that there would be any need to...).

I would suggest you do that which has all the features you need and you are most comfortable with. Personally, I would suggest ESXi, because I value predictability and stability and think that such systems should be running many years without the need to change the base system (aside from security updates of course). If I had to choose KVM, I would therefore choose a distribution with long-term support of 5 to 7 years.