I'm having a very peculiar problem getting SSH to work on my Linux box. I have a passwordless git user identified by an OpenSSH key. If I attempt to ssh into it from either the same or a different linux VM on the network, it fails (see below for full debug info).
But now, here's the weird part: I'm able to ssh in just fine from my Windows 7 box, using the exact same keys! This suggests to me that we may be looking at a client-side problem. If the keys on the server were somehow borked, I shouldn't be able to connect from anywhere.
I've been grinding my wheels in the mud on this for days. There's a ton of topics on the matter, but none of the solutions either worked or applied.
Common solutions I've already tried:
The ~/.ssh permissions have all been properly set on both client and server.
Specifically, ~/.ssh/* was set to 600 (one thread recommended changing authorized_keys (server) to 644, but that had no effect).
The ~/.ssh directory itself was set to 700.
Everything in ~ is owned by the eponymous user/group.
Client (/home/kris/.ssh):
drwx------ 2 kris kris 4096 Apr 11 01:17 . drwx------ 38 kris kris 4096 Apr 11 01:29 .. -rw------- 1 kris kris 458 Apr 11 16:22 config -rw------- 1 kris kris 1675 Apr 10 22:29 id_rsa_kriscraig_git -rw------- 1 kris kris 1675 Apr 11 01:08 id_rsa_kriscraig_git_CRAIGCOM-LINUX -rw------- 1 kris kris 400 Apr 11 01:08 id_rsa_kriscraig_git_CRAIGCOM-LINUX.pub -rw------- 1 kris kris 1675 Apr 11 01:09 id_rsa_kriscraig_git_CRAIGCOM-UT3 -rw------- 1 kris kris 400 Apr 11 01:09 id_rsa_kriscraig_git_CRAIGCOM-UT3.pub -rw------- 1 kris kris 951 Apr 10 22:29 id_rsa_kriscraig_gitolite_Kris -rw------- 1 kris kris 214 Apr 10 22:29 id_rsa_kriscraig_gitolite_Kris.pub -rw------- 1 kris kris 381 Apr 10 22:29 id_rsa_kriscraig_git.pub -rw------- 1 kris kris 1626 Apr 11 17:50 known_hosts
Server (/home/git/.ssh; it's a bit cluttered now due to all the trial/error I've been doing):
drwx------ 2 git git 4096 Apr 11 01:36 . drwx------ 8 git git 4096 Apr 9 17:55 .. -rw-r--r-- 1 git git 2174 Apr 11 01:40 authorized_keys -rw------- 1 git git 904 Aug 4 2012 authorized_keys_bak -rw------- 1 git git 354 Aug 4 2012 authorized_keys_bak2 -rw------- 1 git git 2174 Apr 11 01:13 authorized_keys_bak3 -rw------- 1 git git 160 Apr 10 00:32 config -rw------- 1 git git 1675 Aug 3 2012 id_rsa -rw------- 1 git git 1675 Apr 11 01:08 id_rsa_kriscraig_git_CRAIGCOM-LINUX -rw------- 1 git git 400 Apr 11 01:08 id_rsa_kriscraig_git_CRAIGCOM-LINUX.pub -rw------- 1 git git 1675 Apr 11 01:09 id_rsa_kriscraig_git_CRAIGCOM-UT3 -rw------- 1 git git 400 Apr 11 01:09 id_rsa_kriscraig_git_CRAIGCOM-UT3.pub -rw------- 1 git git 951 Apr 10 00:32 id_rsa_kriscraig_gitolite_Kris -rw------- 1 git git 214 Apr 10 00:33 id_rsa_kriscraig_gitolite_Kris.pub -rw------- 1 git git 381 Aug 3 2012 id_rsa.pub -rw------- 1 git git 414 Aug 4 2012 known_hosts
Though this one's probably obvious, I have, in fact, verified that the file paths in the config are correct.
I've tried generating/distributing new keys multiple times.
I tried copy/pasting the ones already working on Windows as well, even going so far as to use dos2unix to make sure we weren't dealing with any encoding issues (CRLF/LF, etc).
I generated the keys using the standard approach:
ssh-keygen -t rsa
I've tried playing around with the parent home directories' permissions to no avail.
I tinkered with the local configs, as well as /etc/ssh/*_config
And yes, I restarted sshd every time. I even rebooted the server at random intervals, just in case.
I'm sure I'm missing at least a few things. I haven't slept much lately....
I'll take any suggestions at this point. I won't downvote you if it turns out I already tried it. =)
Basic Client/Server Info
Server
- CentOS 5.9 64-bit (VirtualBox)
- 4 GB RAM
- 200 GB HDD (dynamic allocation)
- 4 CPUs
- Bridged networking (they all connect to the same router)
- Tests: N/A
Linux Client #1
- CentOS 5.9 64-bit (VirtualBox)
- 1 GB RAM
- 15 GB HDD (dynamic allocation)
- 1 CPU
- Bridged networking
- Tests: FAIL
Linux Client #2
- The server itself. Seemed a good way to rule-out a few possibilities.
- Tests: FAIL
Windows Client
- Windows 7 Ultimate 64-bit (host for VirtualBox)
- 32 GB RAM
- 200 GB HDD
- 731 GB HDD
- 232 GB HDD
- 465 GB HDD
- 2.72 TB HDD
- 16 CPU
- Tests: SUCCESS
Debug Info (sensitive data redacted)
Server
Resultant entries for two failed ssh attempts in /var/log/secure:
Apr 11 22:21:45 CRAIGCOM-LINUX sshd[14738]: Connection from (WAN IP) port 35326
Apr 11 22:21:45 CRAIGCOM-LINUX sshd[14739]: Connection closed by (WAN IP)
Apr 11 22:21:52 CRAIGCOM-LINUX sshd[14741]: Connection from (WAN IP) port 35328
Apr 11 22:21:52 CRAIGCOM-LINUX sshd[14742]: Connection closed by (WAN IP)
Linux Clients (essentially the same for both)
[kris@CRAIGCOM-LINUX .ssh]$ ssh git@CRAIGCOM-LINUX -vvv
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /home/kris/.ssh/config
debug1: Applying options for CRAIGCOM-LINUX
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to (SERVER HOST) [(SERVER IP)] port 22.
debug1: Connection established.
debug1: identity file "/home/kris/.ssh/id_rsa_kriscraig_git_CRAIGCOM-LINUX" type -1
debug1: loaded 1 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 118/256
debug2: bits set: 484/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /home/kris/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug3: check_host_in_hostfile: filename /home/kris/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host '(SERVER HOST)' is known and matches the RSA host key.
debug1: Found key in /home/kris/.ssh/known_hosts:1
debug2: bits set: 522/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: "/home/kris/.ssh/id_rsa_kriscraig_git_CRAIGCOM-LINUX" ((nil))
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-with-mic,password
debug3: preferred publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred:
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: "/home/kris/.ssh/id_rsa_kriscraig_git_CRAIGCOM-LINUX"
debug3: no such identity: "/home/kris/.ssh/id_rsa_kriscraig_git_CRAIGCOM-LINUX"
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-with-mic,password).
This is where I'm stuck
As you can see from the log, it's returning "no such identity" after attempting to open the private key file. I think this rejection is happening on the client-side. As far as I can tell, it's not sending anything to the server, whatsoever; just opening and abruptly closing the connection.
For the life of me, I can't figure out why it's barfing on the key files on both CentOS boxes! The permissions are set perfectly, the files are readable, neither server is using SELinux, the same keys are working perfectly fine from the Windows client, etc.
I'm good wearing the sysadmin hat; but at the end of the day, I'm a developer, not an IT person. This level of SSH debugging has taken me way outside my area of expertise. I hate to say it, but I've simply run out of ideas. According to every single thing I've been able to find on the internets, this should be working. But it's not. What am I missing?
Thanks for your help! Hopefully, one of you will recognize this issue and boot my ass in the right direction. If you need any further details, please feel free to ask.
--Kris