I always label an old Firefox application as, say FireFox 3.6.17 in the applications folder prior to installing the newer version of Firefox. After that, I install the newer version and rename that application as, say FireFox 12.0.1.
That way, when you see the dialog box saying that another instance of Firefox is already open, pull up the Mac Dock at the bottom of your screen, and you'll see a second Firefox icon in the dock where there wasn't one before. Right-click on the older icon (if you hover the mouse over the icon, you'll see either "Firefox 3.6.17" or the newer one) and go through Options and deselect "Open at Login." Then force-quit the older application, and that icon will go away. Gone. Somehow, when you "threw" or removed the older icon from the dock after upgrading, it didn't change the login status of that icon and kept trying to run both versions of Firefox at login. Problem solved.
When you upgrade and label accordingly, ALWAYS turn off the Login status of the older icon before removing that icon and replacing it with the new one.