I'd like to be able to work with a poorly-behaving application without using a VM (becuse it requires a whole extra license). Does Windows have something like BSD's jails or Linux's containers?
You might want to try WinJail:
WinJail is full implementation of chroot, with additional features like "copy-on-write" mechanism applied to chroot'ed files, additional registry based chroot, and more.
I think "Remote Desktop Services" (see http://technet.microsoft.com/en-us/windowsserver/ee236407 ) is what you are looking for but I think it exceeds your budget :)
Another possibility is turbo.net. It allows to run applications in their own sandbox, preventing it from containing you're system with files, registry entries etc.
If you need if for a your application, you probably want to start with a 'clean' machine. Like
turbo run --mount=C:\FolderWithUntrustyApp clean C:\FolderWithUntrustyApp\untrusty.exe
You also run the installer in it.
turbo run --mount=C:\FolderWithUntrustyApp clean C:\FolderWithUntrustyApp\installer-untrusty.exe
Note: It's not designed to contain things like viruses and advanced malware, which exploit kernel bugs or