You can navigate to
about:config and set the value
1. You will need to restart the browser.
As others have suggested, the close tab button is removed if the tab becomes too small (narrow). This setting determines the minimum width of the tab in pixels below which the close tab button is not shown (
140 is default). Setting it to
1 essentially means always show the close tab button, no matter how narrow the tab is. According to the linked article,
0 is not a valid number (but the button is 16 px anyway).
This setting has an effect in all versions of Firefox from 1.6a1 onwards (confirmed working in 13.0.1).
As of Firefox 23.0, this method no longer seems to work. They may fix it in a future update to the extension, so I will leave it here for now. The config option described above is still available.
An alternative (and the one that I am using) is the Tab Mix Plus extension. This extension includes many other possibly (un)desired features, and obviously isn't as clean as the setting above, but it is what I've been using (for other reasons, the close button was just a side effect).
In TMP's options,
Close tab button >
Click for full size
Wilson's answer links to an extension that is no longer available. Both the config and TMP setting described in this answer achieve the same effect.