I am encrypting a volume with TrueCrypt. I have decided to use
AES as the encryption algorithm, seeing as it is the government standard.
But which hash algorithm is better (more secure)?
I am using Mac.
Both SHA-512 and Whirlpool are 512 bit hashes, while RIPEMD-160 is 160 bits. TrueCrypt uses these hashes with PBKDF2 to derive keys. When creating file volumes, Truecrypt uses 1,000 rounds for both SHA-512 and Whirlpool, but 2,000 rounds for RIPEMD-160. I have some software called TCHead that decrypts TrueCrypt headers. It's very simple and straight-forward and demonstrates how and where these hashes are used.
To answer your question, in my opinion, either SHA-512 or Whirlpool would be fine. I would use either and not worry about that too much. Hope this helps.
I question why the NIST committee recognized Serpent as having superior security and still went with Rijndael and Sha-256/512 for use as AES. I wonder when security is the prime issue why the committee didn't pick it with the excuse that it wasn't fast enough. Serpent scored the Highest in security. True, Rijndael was faster but whatever was chosen would have had hardware compatibility baked into the CPU as AES is. I use Serpent and Whirlpool which the latter is approximately the same as SHA-512. On SSD's the speed difference is less then 5% in the 3 desktops with different Intel chips i5/i7 Haswell, Ivy Bridge and Skylake and Laptop Intel i7. To the best of my knowledge there are known backdoors in Serpent/Whirlpool combo and that makes that my decision easy.