If you consistently end up with multiple MacVim icons in your Dock, then you might want to examine how you are starting MacVim. Specifically,
:help macvim-start says
Do NOT call the MacVim binary inside the app bundle to start MacVim.
From the command line, you would normally use either
open -a MacVim filename or
mvim filename1. From the GUI, you can associate MacVim with particular extensions, drag-and-drop files onto MacVim in Finder or the Dock or into an open MacVim window, or you can use the Services menu to start new buffers with selections from other applications. There is also the
mvim:// URL handler (see
None of these methods require any special flags/options whether an existing instance is running or not and should ever create an extra MacVim icon in the Dock.
1 mvim is a script that comes with MacVim; see
:help mvim. Once you have it installed in a directory in your PATH, then you can use it to edit files in an existing MacVim instance (or will launch one if it was not already running).
The “After last window closes:” Quit MacVim option in the General tab of MacVim’s preferences (⌘, or MacVim > Preferences…) may be what you want.
If you really need something more specific, then you can use the MacVim-custom command
:macaction terminate: to ask the whole MacVim application to exit at any scriptable moment. It should be equivalent to ⌘Q and the MacVim > Quit MacVim menu item (they all send the
terminate: action to the application’s first responder); if you have modified buffers, then it will ask you if you want to “Quit anyway?”.
It sounds like you might want to hook it up to the VimLeave event through an autocommand:
autocmd VimLeave * macaction terminate: