Go to C:\Program Files\Windows Live\Messenger and find msnmsgr.exe. Right click it and go to properties. Go to the compatibility tab and check "Run in compatibility mode for:" and select "Windows Vista". Messenger should now display properly and act like you want it to.
If unchecked, it will remain in your task bar, even when you told it to quit.
first, if it's pinned to your taskbar it will always be visible, but if you ALT+Tab it won't pop up, since it's not active.
second, if your logged in, Live Messenger will always keep a dummy screen open, because they no longer use a System Tray icon (like on Windows XP). When you ALT+Tab it will always pop-up and there's no way to disable this (without any hacks).
However this second behavior has been removed in the new Beta version of Live, so your easiest solution is to upgrade to that.