Using a Distributed or Clustered File System on hosts for Virtual Instances - how?

Question

I'm looking for a way to HOT back up (externally, via USB or LAN) my virtual machines while the machines are still active & running. I have number of challenges:

1. Dealing with large variety of GUEST OS'es ranging from Windows 2000 - 2008 R2, Centos, SUSE, RedHat, Ubuntu, etc.
2. Dealing with multiple physical host machines
3. Dealing with large files (obviously) - in the 10's to 100's of GB's
4. Machines need to stay active / running DURING BACKUP PROCESS.

Ideally this would be like RAID where data is made redundant, but one of the "drives" is external to the host machine. Maybe?

Each of the host machines currently has internal RAID, but that's not enough in the event that the entire machine gets wiped out because there's no external backup. Furthermore it becomes a problem because each host machine might have a different configuration - CPU / RAM, motherboard, etc.

After a great deal of research (although I'm still relatively fresh on the DFS / storage scene), it seems like I'm looking at creating some kind of clustering / distributed file system environment?

Any pointers would be very helpful.

Edit: I intentionally did not mention host or VM platform as I'm open to the right solution that will work for the needs described. I have experience with Linux and Windows, multiple VM platforms. Don't really care about cost at the moment. Trying to understand what solutions are available.

Answer 1

There are several obstacles you're facing:

Ensuring that your backups are consistent

On the filesystem level, this is pretty easy. What I do is have all virtual machines storage on LVM volumes (on Linux), and during backup create a snapshot of the relevant volume and backup that snapshot (mount the filesystem temporarily on the virtualization host, copy files, etc.) I'm using dirvish with some custom scripts for backup.

On the application level, it is impossible to insure data integrity without cooperation of the relevant application. If you have a database running inside your VM, you need to instruct that database to flush data to disk, lock tables, create your LVM snapshot and then unlock the database. No need to keep locks during backup since you're backing up the snapshot, which is atomic. Obviously, this step cannot be generalized, and you need to carefully evaluate what's running inside each of your VMs. This problem is the most serious one, and clustering, distributed filesystems, etc., are not going to help you here, unfortunately.

Managing storage

The backups are going to be pretty expensive in terms of network traffic, if you want all machines backed up into one place. I'm using locally connected eSATA hard disks in each virtualization host (I have just 2). I keep one set of disks offsite, and other connected to hosts, and rotate them regularly.

Answer 2

All the major VM vendors provide solutions to back up VMs while they are running.

For instance, if you run vSphere and use VMWare's VDR (or even Veeam) the hypervisor will quiesce the file system while keeping the VM running. You may notice some sluggishness on the VM, but it's services will not stop during a back up. Obviously you're going to want to follow the recommended hardware guidelines to ensure full uptime.

If you use the VMWare solution, you can use iSCSI targets to backup your data. So, for example, you can buy a cheap netgear NAS (~$700) and backup machines to that and carry them off site. You can also use real-time backup software that vendors are selling now. Basically it mirrors all changes to an external appliance and then just sends the deltas of the changes.

Some companies (like mine) back up the data over the VPN to an external site. We haven't added any real-time backups yet, but we're working with our vendors to get that all worked out. We're not a company that likes to send our data to the cloud.

Clearly there are mission critical systems that run on VMs in the enterprise that cannot go down and need to be backed up. Every vendor is going to have a solution for this.