I structured my private/public-keys in a subfolder of ".ssh" called "keys". Mapping is done by a .ssh/config file. Now I have to retype the passphrase for every key every time I use it.
Before this change, the unlocked key was remembered automatically on demand as default behaviour (OS: Ubuntu).
Use keychain, a simple ssh-agent wrapper, to handle this for you.
Add to your ~/.bashrc:
eval $(keychain --clear --eval id_rsa id_dsa id_ecdsa other.id_rsa )
This way you only need to type your passphrases once upon login. You can still manually add and list keys directly using the agent as usual:
$ ssh-add -l
4096 10:93:fe:6c:2b:36:xx:be:78:xx:bd:xx:b7:f9:0f:46 /home/user/.ssh/id_rsa (RSA)
1024 86:dc:ed:e0:c8:f4:99:bc:99:ee:55:xx:ac:51:e6:cb /home/user/.ssh/id_dsa (DSA)
521 28:xx:f4:d6:10:xx:40:4b:cd:xx:ef:e2:fc:8f:0b:xx /home/user/.ssh/id_ecdsa (ECDSA)
4096 5d:5a:xx:41:xx:2e:ab:ae:xx:ff:5c:47:xx:3a:cf:xx /home/user/.ssh/other.id_rsa (RSA)
The --eval argument in keychain(1) accepts full paths, so you can have your keys wherever you want to.
ssh-add? The desired behaviour is that the passphrase is requested when the key is required (on demand) and stored for following tasks.
keychain is an ssh-agent wrapper as noted above, so it is a convenient way to not have to use ssh-agent directly. Advantages are having an ssh-agent (and a gpg-agent if there's the need) handled for you. You can start with some keys automatically added at login, and add the rest on demand using ssh-add. This solution, coupled with a ~/.ssh/config file specifying which keys to user per host/user is what I've found more comfortable to use.