Ironically, it is probably Firebug which is slowing down Google Maps for you so much. To increase performance, the easiest thing would be to disable Firebug when you're not doing development.
Unfortunately, often times disabling such a useful tool is not an acceptable solution. Thankfully it's only a few components of Firebug that play the biggest role in slowing things down. The biggest performance gains come from disabling the "console" and/or "net" tabs, either by clicking them and deselecting enabled, or going through the options tab. Firebug's performance hit is most noticeable on Google Maps because of the large amount of requests it makes and the complex DOM structure.
If that doesn't help, you can try updating Firefox and Firebug and hope that they've improved performance in recent versions. The most recent version as of when I am writing this is Firebug 1.10.0a7 for Firefox 12, and on my machine it seems to perform reasonably well even with net and console enabled.