NTFS is in general a more robust file system, but the main obstacle with FAT32 that I run into is the 4GB file size limit - you can't have a single file larger than 4GB, which rules out certain things like DVD images or HD movies, etc.
Apart from that, NTFS is a journaled FS and generally more resilient to corruption and fragmentation. (There are a whole host of additional features that NTFS adds, from hard links to better access controls, but they aren't always relevant for your type of usage.)
http://en.wikipedia.org/wiki/NTFS has quite a bit more detailed information.
Most of this isn't likely to matter if you're careful and keep your data backed up; I use FAT32 on at least one external hard drive and haven't had any issues apart from the file size thing, but I also don't use it for long-term storage of anything.