I have confirmed this running Mac OS X 10.6.8 within VMware 5.0.1 on Mac OS X 10.8.3 on a rMBP. Screen capture does not work when using the keyboard shortcuts nor does it work when using the screencapture command line utility.
A workaround is to move your cursor outside the guest VM window and fire Cmd-Opt-4, then space bar, hover over the VM window, and click to screen capture the VM window. I find Cmd-Opt-4 is almost always preferable to 3.
Interestingly, screen capture works fine for me when running 10.7.5 within VMware. I needed to run Office X in order to run Entourage X. Office X requires Rosetta so I had to run 10.6. From there I was able to diagnose that the only version of SSL that Entourage X supports is SMTPS.
Running Mac OS X 10.5 and 10.6 Server under VMware is permissible under Apple's licensing. They are both presented as valid options when creating a new VM with VMware Fusion 4 and 5 (and perhaps prior too). A 10-user OS X server license can be had for under $50 on eBay. For a few cents more you can get an Xserve included with the license.
