This is not a good technique. Relying on malware to behave nicely because it might be under the microscope is a bit like relying on cats to stay put because you told them to. It's an interesting idea, but one which is not worth implementing as an anti-malware solution.
That said, as Marc suggested - just actually run your OS in a VM or hypervisor, if you want malware to behave itself as if it is in a virtualized environment. The performance hit is the tiny price you pay for such enhanced peace of mind.
One other item of note is that there are a fair number of legitimate desktop apps which don't work under VMs because their DRM thinks they might be in the process of being reverse engineered. The usability hassle from that would be terrible.