SSH agent forwarding is not working

Question

My SSH agent forwarding is not working. The answer from this question is quite old and the answer is conflicting with the comments. Can we have an updated answer please?

Basically I can SSH from computer A to computer B, or from computer A to computer C, but not from computer A to computer B, then computer B to computer C. When connecting from computer A to either computer B or computer C, I always use ssh -A to force SSH agent forwarding. But I’m still not able to connect computer A -> computer B -> computer C without being asked for a passphrase.

echo "$SSH_AUTH_SOCK" works both on the server and my local machine, but on SSH from A -> B, ssh-add -l yields:

The agent has no identities.

Whereas it lists two keys on A.

All three machines are configured with the standard Ubuntu SSH configuration without any changes, the content of /etc/ssh/ssh_config and /etc/ssh/sshd_config are posted at http://paste.ubuntu.com/11482964/.

EDIT: as per request,

Here is the log of -Avvv from B to C:

OpenSSH_6.2p2 Ubuntu-6ubuntu0.1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /home/myid/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to boxc.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/myid/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/myid/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-1024
debug1: identity file /home/myid/.ssh/id_rsa-cert type -1
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/myid/.ssh/id_dsa" as a RSA1 public key
debug1: identity file /home/myid/.ssh/id_dsa type 2
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: identity file /home/myid/.ssh/id_dsa-cert type -1
debug1: identity file /home/myid/.ssh/id_ecdsa type -1
debug1: identity file /home/myid/.ssh/id_ecdsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2p2 Ubuntu-6ubuntu0.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2p2 Ubuntu-6
debug1: match: OpenSSH_6.2p2 Ubuntu-6 pat OpenSSH*
debug2: fd 3 setting O_NONBLOCK
debug3: put_host_port: boxc
debug3: load_hostkeys: loading entries for host "boxc" from file "/home/myid/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:15
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],ssh-rsa,[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],zlib,none
debug2: kex_parse_kexinit: [email protected],zlib,none
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found [email protected]
debug1: kex: server->client aes128-ctr [email protected] [email protected]
debug2: mac_setup: found [email protected]
debug1: kex: client->server aes128-ctr [email protected] [email protected]
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: RSA ed:26:20:93:4c:88:ef:17:70:e3:d4:7a:42:4c:8e:69
debug3: put_host_port: [192.168.2.122]:21
debug3: put_host_port: boxc
debug3: load_hostkeys: loading entries for host "boxc" from file "/home/myid/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:15
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries from file "/home/myid/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:16
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'boxc' is known and matches the RSA host key.
debug1: Found key in /home/myid/.ssh/known_hosts:15
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/myid/.ssh/id_rsa (0x7f7e....e760),
debug2: key: /home/myid/.ssh/id_dsa (0x7f7e....e7a0),
debug2: key: /home/myid/.ssh/id_ecdsa ((nil)),
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/myid/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 149
debug2: input_userauth_pk_ok: fp 22:32:...:1d:e3
debug3: sign_and_send_pubkey: RSA 22:32:...:1d:e3
debug1: key_parse_private_pem: PEM_read_PrivateKey failed
debug1: read PEM private key done: type <unknown>
Enter passphrase for key '/home/myid/.ssh/id_rsa': 

Answer 1

A plausible explanation is as follows: when you ssh from A to B, a shell startup script that gets sourced on B starts an agent local to B.

In case of Bash being your shell on B, check your .bashrc on B.

Suppose the script properly starts an agent local to B. This means it overwrites the relevant environment variable(s) ($SSH_AUTH_SOCK essentially) in the shell on B. In effect anything that tries to use an agent in this shell session will find the agent running on B that knows nothing about your keys on A. When you ssh from B to C, the agent on B has no identities, you are asked for the passprhase for key /home/myid/.ssh/id_rsa stored on B.

If you knew the previous value of $SSH_AUTH_SOCK on B, you could locate the socket on B that is still connected to the agent running on A. If you (manually) changed $SSH_AUTH_SOCK back to the previous value, things would start working as you expect. My point is the forwarding itself is not broken, the socket provided by the SSH server on B is there and it works. The issue is your processes on B cannot find it because they find and use the wrong socket instead, due to an agent on B having been started and $SSH_AUTH_SOCK having been changed.

You can fix this by not allowing the startup script on B to run a new agent. Responsible code is there probably for a reason, so deleting it is not necessarily a good idea. Ask yourself what the reason is. If there is a reason to keep the code then a sane solution will be to make the script check if $SSH_AUTH_SOCK expands to an empty string and only then set a new agent up.

I cannot be sure if this is what happens in your particular case; still I think the scenario is plausible.