If you're referring to the internal "Search for.." of Firefox, probably there's no such option.
Well, in fact I'm surprised you want that behavior, but of course it's a matter of personal taste and work flow. But think of 'heavy' pages you're on -- when you search something and it opens in the same tab, and then you click 'Back', it will be a lot of work for the browser to re-render the page. The work flow is also different for me, because I often search not in Google, but look up words in dictionaries using Context Search add-on.
There's kind of similar option browser.search.context.loadInBackground recently added, saying whether to focus the new tab or not. Perhaps you should open an issue in Bugzilla.
Instead of right clicking and choosing 'Search in Google', you can
- make sure Google is selected as default search engine in the search field you have in top-right of the browser,
false (this affects only search from the search field),
- select the text,
- drag it from the page onto the search field of the Firefox.
That way you'll have it opened in the same tab.