How can I run a single Mac app inside a virtual machine without installing Mac OS X?
You can't. You simply cannot run just a Mac app in a VM without installing OS X. Installing OS X in a VM is also against the Mac OS X Software License Agreement where it is stated:
"you are granted a limited non-exclusive license to install, use and run one (1) copy of the Apple Software on a single Apple-branded computer at a time.
Which rules out VMs. You could of course try to find a cross-platform alternative and use that.
However, you might want to look into OS X's support for application sandboxing: