Make sure you have the latest version of SetPoint. You may need to uninstall and reinstall the SetPoint software. By default the middle button will be set to zoom or the application switcher, but you should be able to use SetPoint to configure the middle button as "Other -> Middle Button". Then Firefox will recognize it correctly and close the tab.
Also, the new setpoint software will let you do custom actions for different applications on some mice, so make sure that when you configure it you select the "All Other Programs" or create a special managed program for Firefox.
You may also need to check under the Mouse options in the control panel to see if there are any other configuration issues. If that doesn't work make sure try uninstalling any other mouse software (Microsoft IntelliPoint, uberOptions, Synaptics Touchpad, etc) as well as uninstalling any drivers listed under the "Mice and other pointing devices" in the device manager. Then reboot and it should reinstall the drivers.
Finally if that doesn't work you need to figure out if the problem is in the Mouse, in Set Point, or in the OS. Try this:
Plug in another 3 button mouse that is not made by Logitech and see if the middle button works. If it still doesn't work try using something like Mousekeys to simulate the middle mouse button. If none of these work it may be an OS issue.
Try alternate mapping software such as X-Mouse. Also try mapping the middle click function to another button to see if that work.
Try changing to a different driver. Replacing the existing driver with an older version or with a generic mouse driver might get it to work again. Also try it on another computer and see if works there.
Buy a different mouse :-P Sometimes it is not worth fixing if you can just replace it. I'd highly recommend the MX Revolution, since it has lots of buttons and a nice "free spin" scroll wheel, but it is not a trackball.