The problem is that Windows Explorer also provides the whole shell for Windows. So when you click that Folder icon, you're most likely already using Explorer. It just starts another process (or window).
You might want to look into shell replacements.
Ok, let me go into a little bit more detail about my understanding of things.
How the whole system works
Let's start with my desktop.
My desktop is drawn by the shell. The shell is provided by the initial
explorer.exe process that runs when Windows starts. If I have a folder on my desktop:
I can double-click it to open it:
This created a new
explorer.exe process which is now displaying the window that shows the contents of my folder. But there are many way the same window could have been opened.
So why did Windows to decide to use
explorer.exe to display that folder?
Well, as I said, the desktop is already an
explorer.exe process. So it will simply use
ShellExecuteEx() to let the shell decide what to do with it. The shell will then create a new
explorer.exe instance (because it is hard-wired to do so to my knowledge).
So what you can do, is not use
explorer.exe to provide your shell (as you already seem to do). You can simply provide an alternative application in the registry under:
But nobody is stopping that new shell from calling
ShellExecuteEx(), which would, again, invoke the default shell and, thus,
So, what now?
So if you really want to replace Windows Explorer, replace the file
explorer.exe. Because, in the end, every program could still try to execute
C:\Windows\explorer.exe directly and then you have the same old Explorer again.
But, please keep in mind, I have never attempted to replace
explorer.exe on a system, so I don't know if that would even be possible and/or stable to use.
How does application XYZ do it?
There are several applications that simply plug into Explorer, like QTTabBar. Maybe Explorer++ works the same way. This way you don't need to touch anything as your modification is always loaded into new
explorer.exe processes. Going into detail about that process would take a whole new answer.
What about the File-Open dialog?
This is a whole new set of problems. The File Save and File Open dialogs are part of the Common Dialogs Library. Trying to replace those is hard to imagine. Pretty much the same can be said about those as I said about
explorer.exe. But this would be far more scary.