Set up ssh-agent (or equivalent), then enable agent forwarding for SSH connections.
Enable ssh-agent on your local machine
"Local" being the one you're sitting at. You should not need to run the agent anywhere else.
If you're lucky, one will be already running; most Linux distributions start an instance for every X11 session. Run ssh-add to import your keys; ssh-add -l to list keys.
If using GNOME or Mac OS X, you will probably have their own agent implementation, which imports known keys on login. With the standard OpenSSH ssh-agent, though, you have to do this manually (although scripts and PAM modules exist to automate).
On Windows, PuTTY comes with Pageant, and Cygwin OpenSSH with the same ssh-agent (although much more complicated to use than in Unix).
Enable agent forwarding
Once you have the agent working and keys loaded, try to connect like this:
A> ssh-add -l
A> ssh -A B ssh-add -l
The -A option enables agent forwarding over the SSH connection, so if everything works right, both commands above should give identical output.
$ ssh-add -l
2048 06:b7:29:1e:6b:ea:c3:04:c4:33:fc:48:e0:62:2f:ef grawity@anywhere (old key) (RSA)
4096 19:3c:f0:79:32:00:fa:04:2f:15:5d:2a:e2:c9:a3:ad grawity@anywhere (2011-11-01) (RSA)
$ ssh panther ssh-add -l
Could not open a connection to your authentication agent.
$ ssh -A panther ssh-add -l
2048 06:b7:29:1e:6b:ea:c3:04:c4:33:fc:48:e0:62:2f:ef grawity@anywhere (old key) (RSA)
4096 19:3c:f0:79:32:00:fa:04:2f:15:5d:2a:e2:c9:a3:ad grawity@anywhere (2011-11-01) (RSA)
If it works, make it persistent by editing ~/.ssh/config – put this at the bottom:
Host *
ForwardAgent yes
Instead of * you can give a list of trusted names; see manual page of ssh_config for exact syntax.
You can enable forwarding on all machines; there is no limit to how much you can chain.