If you're referring to the internal "Search for.." of Firefox, probably there's no such option.
Well, in fact I'm surprised you want that behavior, but of course it's a matter of personal taste and work flow. But think of 'heavy' pages you're on -- when you search something and it opens in the same tab, and then you click 'Back', it will be a lot of work for the browser to re-render the page. The work flow is also different for me, because I often search not in Google, but look up words in dictionaries using Context Search add-on.
There's kind of similar option browser.search.context.loadInBackground recently added, saying whether to focus the new tab or not. Perhaps you should open an issue in Bugzilla.
Alternative workaround:
Instead of right clicking and choosing 'Search in Google', you can
- make sure Google is selected as default search engine in the search field you have in top-right of the browser,
- set
browser.search.openintab to false (this affects only search from the search field),
- select the text,
- drag it from the page onto the search field of the Firefox.
That way you'll have it opened in the same tab.