This happened before. And I fixed it with an about:config hack.
if you open about:config in your address bar, you can change the setting manually. Change: browser.link.open_newwindow Mine was set to 3, which told the popups to always open in a new tab.
Niceness. It's a pretty easy fix. Set it to 2. I just wonder what changed it to 3. Wasn't me.