I have a 64-bit OS, but I seem to remember reading that Office 64 has issues with various plugins, etc.
Anyone have a recommendation for when I should install the 64 bit vs. when I should install the 32 bit?
That's the compatibility issue you raise in your question. The only real advantage to the 64-bit version is that you can work with more than 2GB of data (typically relevant to Excel). If you are working with truly enormous amounts of data and have run up against this limit (or have reason to think you soon will) then consider 64-bit. Otherwise I'd stick with 32-bit.