Extensions don't actually change what type of file something is, although they often do hint to the system what do do with a particular item.
DMGs are not executables, they are disc images. They are files that act as discs, which often contain the executables and other files for applications. When you double-click on one, the system mounts it and you then have access to the files inside.
Both Windows and OS X execute binary executable files, although they are in different formats. The big difference in what a user sees if using the GUI is that on Windows, you usually see the .exe executable file, which resides in a folder that contains other files and information that the program uses. On OS X, all those other files, along with the primary executable, are contained in a "bundle" (really a folder) with the extension .app. Double clicking on a .app bundle will execute the internal executable and load your application.
Outside of the GUI, in the actual file structure, Mac and Windows systems are really quite similar; these differences are mostly a question of how the system presents things to the user for manipulation in the GUI.