On Windows (and possibly other 64-bit operating systems), 32 bit applications are supported. One possible advantage of a 32 bit application is therefore greater portability.
The main advantage of a 64 bit application is that it can access 4GB or more of memory as a flat address space. However, relatively few applications benefit from that. The disadvantage of a 64 bit application is that every pointer takes up 8 bytes of memory instead of 4 bytes. This means that program code and data structures take up more memory, which means they can be slower. How much slower varies a lot. It's possible for a 64 bit application to be a tad faster, due to memory alignment issues, though my guess is that the larger memory footprint will outweigh this for a while.
As processors get more optimised for 64 bit code, and less optimised for 32 bit, over time - and as the savings get less relevant compared with the hassles - these issues will become as irrelevant as the potentially slightly better efficiency of 16 bit code for applications that only need 64K of memory. To some degree that's already happening - but as long as lots of people keep hanging on to Windows XP, the portability issue will remain.
On Linux, "old 32 bit apps" are virtually a non-issue. Because source code is available for most software, either a 32 bit or 64 bit compile is usually just as easy to get, even for old apps. If you're using 64 bit Linux, all your apps are probably compiled as 64 bit code - I haven't checked whether any 32-bit binary support is included in 64-bit Linux even though I use 64-bit Linux every day, because it's a non-issue.