7

First of all, I've checked all the following

but none of them helped. So, here is my question:

I can ssh from A to B, or from A to C, but not from A to B, then B to C. When connecting from A to either B or C, I always use ssh -A to force SSH Agent Forwarding.
But why I'm still not able to connect A -> B -> C without being asked for passphrase?

UPDATE: nearly three years later, the same problem still haunts me, but now I've narrowed down the problem a bit:

Symptom: I can ssh A -> B, or A -> C, but not A -> B -> C, or A -> C -> B.

The problem is exactly described by the topic -- SSH Agent Forwarding is not working.

From

Troubleshoot SSH issues
https://confluence.atlassian.com/bitbucket/troubleshoot-ssh-issues-271943403.html

it says:

To list your loaded keys, enter ssh-add -l. If you don't see the SSH key you want to use...

then there is problem -- the SSH key you want to use is NOT loaded.

This is what's happening when I'm at A -> B or A -> C. I.e., right after I did ssh -A into an intermediate server. the SSH key is lost, not forwarded and not loaded.

$ ssh-add -l
The agent has no identities.

That's the reason why I cannot ssh further without the passphrase.

It does have SSH_AUTH_SOCK variable setup and several ssh-agent around though:

$ echo "$SSH_AUTH_SOCK"
/tmp/ssh-RtEuLOmFDBet/agent.3722

$ ps -e  | grep [s]sh-agent
 3723 ?        00:00:00 ssh-agent
 4613 ?        00:00:00 ssh-agent

It does not seem to be related to my own environment as they are identical, or the /etc/ssh/sshd_config file, as I've compared the ones from the intermediate servers that are working or not working.

UPDATE END.

More info: all three machines, are configured with the standard Ubuntu ssh configuration. I.e., the AllowAgentForwarding option is not in /etc/ssh/sshd_config, though I doubt whether it should, because I saw "Since agent forwarding defaults on, it should be enough to remove any AllowAgentForwarding line from sshd_config." from Extra configuration required for ssh-agent forwarding?.

Some say ssh-add will do, but when I do it on B or C, it asks me to Enter passphrase for my .ssh/id_rsa. Some say check the SSH_AUTH_SOCK, but I do have it on B or C (either from A to B, or from A to C):

$ env | grep SSH_AUTH_SOCK
SSH_AUTH_SOCK=/tmp/ssh-RTScJ5PZh9Mh/agent.2083

Is the Agent Forwarding not working because of missing the AllowAgentForwarding option ? Then which one (A, B, or C) should I put it in? Wouldn't ssh -A be enough? Also I have the .ssh/id_rsa files on both B and C, is that the reason ssh-add ask for the passphrase for them?

EDIT:

Here is the log of -Avvv from B to C:

OpenSSH_6.2p2 Ubuntu-6ubuntu0.1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /home/myid/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to boxc.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/myid/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/myid/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-1024
debug1: identity file /home/myid/.ssh/id_rsa-cert type -1
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/myid/.ssh/id_dsa" as a RSA1 public key
debug1: identity file /home/myid/.ssh/id_dsa type 2
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: identity file /home/myid/.ssh/id_dsa-cert type -1
debug1: identity file /home/myid/.ssh/id_ecdsa type -1
debug1: identity file /home/myid/.ssh/id_ecdsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2p2 Ubuntu-6ubuntu0.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2p2 Ubuntu-6
debug1: match: OpenSSH_6.2p2 Ubuntu-6 pat OpenSSH*
debug2: fd 3 setting O_NONBLOCK
debug3: put_host_port: boxc
debug3: load_hostkeys: loading entries for host "boxc" from file "/home/myid/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:15
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5-etm@openssh.com
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com zlib@openssh.com
debug2: mac_setup: found hmac-md5-etm@openssh.com
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com zlib@openssh.com
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: RSA ed:26:20:93:4c:88:ef:17:70:e3:d4:7a:42:4c:8e:69
debug3: put_host_port: [192.168.2.122]:21
debug3: put_host_port: boxc
debug3: load_hostkeys: loading entries for host "boxc" from file "/home/myid/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:15
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries from file "/home/myid/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:16
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'boxc' is known and matches the RSA host key.
debug1: Found key in /home/myid/.ssh/known_hosts:15
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/myid/.ssh/id_rsa (0x7f7e....e760),
debug2: key: /home/myid/.ssh/id_dsa (0x7f7e....e7a0),
debug2: key: /home/myid/.ssh/id_ecdsa ((nil)),
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/myid/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 149
debug2: input_userauth_pk_ok: fp 22:32:...:1d:e3
debug3: sign_and_send_pubkey: RSA 22:32:...:1d:e3
debug1: key_parse_private_pem: PEM_read_PrivateKey failed
debug1: read PEM private key done: type <unknown>
Enter passphrase for key '/home/myid/.ssh/id_rsa': 

I've compared with my good session (A->C), and found all are none different, other than the last 3 lines, which starts with "key_parse_private_pem: PEM_read_PrivateKey failed". The good session instead has:

debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to boxc

Everything else are the same.

Again, my environment:

$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:6.2p2-6ubuntu0.1
  Candidate: 1:6.2p2-6ubuntu0.4
  Version table:
     1:6.2p2-6ubuntu0.4 0
        500 http://archive.ubuntu.com/ubuntu/ saucy-updates/main amd64 Packages
     1:6.2p2-6ubuntu0.3 0
        500 http://security.ubuntu.com/ubuntu/ saucy-security/main amd64 Packages

% sshd -v
sshd: illegal option -- v
OpenSSH_6.2p2 Ubuntu-6ubuntu0.1, OpenSSL 1.0.1e 11 Feb 2013

Thanks

  • Have you tried ssh -Avv (or maybe more -v, I don't know the maximum verbosity level by heart) and watched the output? – ua2b Nov 15 '14 at 1:58
  • yeah, I tried, with 4 v's. But don't know what to look for especially, and found nothing suspicious... Edit: Hold on, found something... – xpt Nov 15 '14 at 2:21
  • 1
    So you ssh-add your key on A, ssh-add -l shows the key has been loaded, then you log into B using ssh -A. Does ssh-add -l still show the same key? You don't need (the same key in) .ssh/id_rsa on B. I assume, your key is encrypted with the password that works well if you enter it at the prompt that you don't want to see? – ua2b Nov 15 '14 at 4:17
  • Yes, all your sayings apply well to me, except that, on B, ssh-add -l does not show the same key any more. In fact I got The agent has no identities. Also I do have the same key in .ssh/id_rsa on B. Thanks again! – xpt Nov 15 '14 at 15:06
4

I've been trying to troubleshoot this problem for nearly seven years, and finally it get solved -- I launch keychain in my ~/.profile, which starts its own 'ssh-agent', even on machine B & C. This is the source of the problem, because keychain's ssh-agent overshadow the sshd provided one.

Removing it (keychain) from my ~/.profile solved the problem.

Update, another possibility, ssh-agent etc. usually get started as part of starting up the GUI on the local system. e.g., in another case, the call's hidden in /etc/X11/xdm/sys.xsession!

I confirm my SSH Agent Forwarding is working by doing, in MachineA,

ssh -t MachineB ssh MachineC

while ssh MachineB then, within it ssh MachineC was failing.

I'll start it (ssh-agent from keychain etc) manually only from machine A from now on.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.