I think you are better off running a host OS that supports your chosen VM well and then tune your host to eliminate everything you don't need. Trust me, the future headaches you will avoid from trying to cram 10 pounds of (well, you know) into a 5 pound bag are worth the price of using a 'less minimalist' distro. Pick a distro you're familiar with. Debian, Ubuntu, Fedora, Mandriva are all good VirtualBox choices.
If you pick a host OS that's on the bloody edge of your vm, you're always going to have pesky issues with remote disks, audio, 3D, peripherals, you name it. It won't be all those problems all the time, but just something annoying that you really don't need in your life.
I run Windows XP SP2 routinely in VirtualBox on Ubuntu and Fedora and I can tell you it's faster virtualized than if I had installed XP on my machine. I have no idea why that is, but it seems to be true.
I can't speak for VMWare.