Does the virtual machine provide an environment which is fundamentally
indistinguishable from a physical machine? Of course, there will be
some practical differences (like hypervisor call escape hatches, dummy
hardware component names, etc.) which allow the detection of a virtual
machine, but will there be any incompatibilities?
Hardware accesses still go through the host OS.
PC operating systems have been decoupled from most hardware for a long time now. Windows and Linux both employ drivers that are the proxy for hardware access. Because of this, it's possible to provide "paravirtualized" drivers that are a proxy to work in a virtual environment, making things a lot easier to implement than emulation - where you'd have to design programs that 100% replicate all the weird and sometimes undocumented behavior of CPUs and hardware.
If the environment is fully compatible with a physical machine, then
are nested virtual machines possible?
The VMXON instruction that "turns on" the CPU virtualization will not work when the CPU is "in" a virtual machine. If this happens, a "VM exit" occurs - meaning control is passed back to the hypervisor - and the host OS must now decide what to do (typically emulating an "illegal instruction" exception). It's possible for the host OS to emulate this instruction and the rest of the CPU virtualization behavior if desired (it would be difficult and slower).
If not, does that mean that the guest OS has to be specifically
adapted for running inside a virtual machine? If so, then does that
mean that most of today's OS'es have already been adapted for most VM
See my first point. Paravirtualized drivers for VMWare and other hypervisors exist. The core OS itself doesn't really have to change unless A) it's something tightly coupled to specific hardware such as DOS and B) it's decided that changing the OS is better than emulating the hardware it expects.
Are these things different for software-based virtualization vs
Software based virtualization is slower and probably less secure.
What exactly is the difference between software-based virtualization
Simplifying a bit, x86 CPUs for a long time have had generally two privilege levels, user mode and supervisor, or kernel mode. Hardware virtualization support adds a privilege level higher than supervisor, i.e. hypervisor. An example of a difference is that the MMU that a CPU's OS uses to manage memory pages is extended to allow hypervisors to work with it. This makes it easier for a hypervisor to allocate and isolate memory for a specific VM. IOMMU is also a hardware feature that also provides a lot of facilities that make virtualization more efficient.