How does the Compatibility Mode in Windows work internally?
Compatibility mode is achieved using so called shims. There is a good article on TechNet describing how these work.
Windows application files contain an import table which tells the application loader which DLLs the application needs and which functions it uses from them. A process might for example reference
GetVersionEx in the
kernel32.dll. When a program shall run in compatibility mode, then the shim is put between the application and the shim replaces the
GetVersionEx function, so that the application does not call
kernel32.dll but the
GetVersionEx in the shim. The shimmed functions then implement the behaviour of previous Windows versions.
GetVersionEx is an easy sample, each Windows version returns its own version numbers in
GetVersionEx, so when faking an old Windows the
GetVersionEx function now not returns the Windows 7 version numbers but for example the Windows XP version numbers. So the application will believe it is running on Windows XP.
There have also been some other changes from Windows version to Windows version. In older versions for example, if a program loaded a DLL, the search path for the DLL also included the current directory. This is a security issue, so newer versions of Windows by default don't search in the current directory. With the proper shim you can simulate the old behaviour.
Since shims are just a layer between the application and the Windows API a shim can just do what the application could do itself. The shim cannot be used for example to circumvent UAC or access protected files.
If you want to know more, here are some links you might find interesting:
Especially Microsoft Application Compatibility Toolkit is worth a look. This tool gives you an overview over the applications with known issues, all available compatibility fixes and modes and which fixes are applied to each application.
I think a lot of different things happen. A straightforward example is that a program might check your windows version, but get confused by the return value of a new operating system. So using compatibility mode would tell windows to report a wrong version. Raymond Chen mentions some more things: http://blogs.msdn.com/oldnewthing/archive/2003/12/23/45481.aspx#45590
My knowledge of Compatibility Mode is that it causes several Windows system calls to lie to the program.
Old-style file paths are also automatically translated when a program in Compatibility mode
refers to a file in well-known system folders.
C:\Documents and Settings is translated to
when running on Windows 7 a program in XP compatibility mode.
This article does a good job of explaining it.
With Windows 7, however, Redmond has provided a solution to the problem: Windows XP Mode. Windows XP Mode uses virtualization technology to let applications running on a virtualized copy of Windows XP show up in the Windows 7 Start menu and on the Windows 7 desktop.