Suppose, I use Debian unstable or experimental version as guest OS. After installing new software it completely breaks the system. Can it also affect host OS?
Depending on the virtual machine, it can be possible.
For example, if the VM software has 3D acceleration enabled, then it gives the guest much more access to your graphics card than you might expect. There also have been several bugs in various virtualization software that would allow the guest to obtain the same privileges as the host.
But although this can be used by malicious software running in the guest, it is very unlikely to happen accidentally. If you merely destroy the guest OS in some way or another, there is no way it can affect the host.
Most of the answers here discuss the virtual machine itself (as does your question) - but the real concern lies with the virtualization software. The concern is that virtualization software has a vulnerability in it's virtualized environment, or doesn't protect the host operating system. Most available virtualization products are pretty solid and have good track records of stability and security.
In normal circumstances, no. The virtual machine is completely separated from the host machine.
But it is sometimes possible to share a folder between the host and the guest, and in that way is it possible to do harmful things to the host. Or you could connect in the guest to the host via internet and do bad things that way. But otherwise, it is quite safe to experiment in a virtual machine.
Note: depending on the virtualisation software, the guest file system is stored in a single file somewhere on the host. But it isn't possible for the guest to easily break out of the sandbox the virtualisation software provides.