13

I'm having a very peculiar problem getting SSH to work on my Linux box. I have a passwordless git user identified by an OpenSSH key. If I attempt to ssh into it from either the same or a different linux VM on the network, it fails (see below for full debug info).

But now, here's the weird part: I'm able to ssh in just fine from my Windows 7 box, using the exact same keys! This suggests to me that we may be looking at a client-side problem. If the keys on the server were somehow borked, I shouldn't be able to connect from anywhere.

I've been grinding my wheels in the mud on this for days. There's a ton of topics on the matter, but none of the solutions either worked or applied.

Common solutions I've already tried:

  1. The ~/.ssh permissions have all been properly set on both client and server.

    Specifically, ~/.ssh/* was set to 600 (one thread recommended changing authorized_keys (server) to 644, but that had no effect).

    The ~/.ssh directory itself was set to 700.

    Everything in ~ is owned by the eponymous user/group.

    Client (/home/kris/.ssh):

    drwx------  2 kris kris 4096 Apr 11 01:17 .
    drwx------ 38 kris kris 4096 Apr 11 01:29 ..
    -rw-------  1 kris kris  458 Apr 11 16:22 config
    -rw-------  1 kris kris 1675 Apr 10 22:29 id_rsa_kriscraig_git
    -rw-------  1 kris kris 1675 Apr 11 01:08 id_rsa_kriscraig_git_CRAIGCOM-LINUX
    -rw-------  1 kris kris  400 Apr 11 01:08 id_rsa_kriscraig_git_CRAIGCOM-LINUX.pub
    -rw-------  1 kris kris 1675 Apr 11 01:09 id_rsa_kriscraig_git_CRAIGCOM-UT3
    -rw-------  1 kris kris  400 Apr 11 01:09 id_rsa_kriscraig_git_CRAIGCOM-UT3.pub
    -rw-------  1 kris kris  951 Apr 10 22:29 id_rsa_kriscraig_gitolite_Kris
    -rw-------  1 kris kris  214 Apr 10 22:29 id_rsa_kriscraig_gitolite_Kris.pub
    -rw-------  1 kris kris  381 Apr 10 22:29 id_rsa_kriscraig_git.pub
    -rw-------  1 kris kris 1626 Apr 11 17:50 known_hosts
    

    Server (/home/git/.ssh; it's a bit cluttered now due to all the trial/error I've been doing):

    drwx------ 2 git git 4096 Apr 11 01:36 .
    drwx------ 8 git git 4096 Apr  9 17:55 ..
    -rw-r--r-- 1 git git 2174 Apr 11 01:40 authorized_keys
    -rw------- 1 git git  904 Aug  4  2012 authorized_keys_bak
    -rw------- 1 git git  354 Aug  4  2012 authorized_keys_bak2
    -rw------- 1 git git 2174 Apr 11 01:13 authorized_keys_bak3
    -rw------- 1 git git  160 Apr 10 00:32 config
    -rw------- 1 git git 1675 Aug  3  2012 id_rsa
    -rw------- 1 git git 1675 Apr 11 01:08 id_rsa_kriscraig_git_CRAIGCOM-LINUX
    -rw------- 1 git git  400 Apr 11 01:08 id_rsa_kriscraig_git_CRAIGCOM-LINUX.pub
    -rw------- 1 git git 1675 Apr 11 01:09 id_rsa_kriscraig_git_CRAIGCOM-UT3
    -rw------- 1 git git  400 Apr 11 01:09 id_rsa_kriscraig_git_CRAIGCOM-UT3.pub
    -rw------- 1 git git  951 Apr 10 00:32 id_rsa_kriscraig_gitolite_Kris
    -rw------- 1 git git  214 Apr 10 00:33 id_rsa_kriscraig_gitolite_Kris.pub
    -rw------- 1 git git  381 Aug  3  2012 id_rsa.pub
    -rw------- 1 git git  414 Aug  4  2012 known_hosts
    
  2. Though this one's probably obvious, I have, in fact, verified that the file paths in the config are correct.

  3. I've tried generating/distributing new keys multiple times.

    I tried copy/pasting the ones already working on Windows as well, even going so far as to use dos2unix to make sure we weren't dealing with any encoding issues (CRLF/LF, etc).

    I generated the keys using the standard approach:

    ssh-keygen -t rsa
    
  4. I've tried playing around with the parent home directories' permissions to no avail.

  5. I tinkered with the local configs, as well as /etc/ssh/*_config

    And yes, I restarted sshd every time. I even rebooted the server at random intervals, just in case.

  6. I'm sure I'm missing at least a few things. I haven't slept much lately....

    I'll take any suggestions at this point. I won't downvote you if it turns out I already tried it. =)

Basic Client/Server Info

Server

  • CentOS 5.9 64-bit (VirtualBox)
  • 4 GB RAM
  • 200 GB HDD (dynamic allocation)
  • 4 CPUs
  • Bridged networking (they all connect to the same router)
  • Tests: N/A

Linux Client #1

  • CentOS 5.9 64-bit (VirtualBox)
  • 1 GB RAM
  • 15 GB HDD (dynamic allocation)
  • 1 CPU
  • Bridged networking
  • Tests: FAIL

Linux Client #2

  • The server itself. Seemed a good way to rule-out a few possibilities.
  • Tests: FAIL

Windows Client

  • Windows 7 Ultimate 64-bit (host for VirtualBox)
  • 32 GB RAM
  • 200 GB HDD
  • 731 GB HDD
  • 232 GB HDD
  • 465 GB HDD
  • 2.72 TB HDD
  • 16 CPU
  • Tests: SUCCESS

Debug Info (sensitive data redacted)

Server

Resultant entries for two failed ssh attempts in /var/log/secure:

    Apr 11 22:21:45 CRAIGCOM-LINUX sshd[14738]: Connection from (WAN IP) port 35326
    Apr 11 22:21:45 CRAIGCOM-LINUX sshd[14739]: Connection closed by (WAN IP)
    Apr 11 22:21:52 CRAIGCOM-LINUX sshd[14741]: Connection from (WAN IP) port 35328
    Apr 11 22:21:52 CRAIGCOM-LINUX sshd[14742]: Connection closed by (WAN IP)

Linux Clients (essentially the same for both)

    [kris@CRAIGCOM-LINUX .ssh]$ ssh git@CRAIGCOM-LINUX -vvv
    OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
    debug1: Reading configuration data /home/kris/.ssh/config
    debug1: Applying options for CRAIGCOM-LINUX
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: Applying options for *
    debug2: ssh_connect: needpriv 0
    debug1: Connecting to (SERVER HOST) [(SERVER IP)] port 22.
    debug1: Connection established.
    debug1: identity file "/home/kris/.ssh/id_rsa_kriscraig_git_CRAIGCOM-LINUX" type -1
    debug1: loaded 1 keys
    debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
    debug1: match: OpenSSH_4.3 pat OpenSSH*
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_4.3
    debug2: fd 3 setting O_NONBLOCK
    debug1: SSH2_MSG_KEXINIT sent
    debug1: SSH2_MSG_KEXINIT received
    debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
    debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
    debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
    debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
    debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
    debug2: kex_parse_kexinit: none,[email protected],zlib
    debug2: kex_parse_kexinit: none,[email protected],zlib
    debug2: kex_parse_kexinit: 
    debug2: kex_parse_kexinit: 
    debug2: kex_parse_kexinit: first_kex_follows 0 
    debug2: kex_parse_kexinit: reserved 0 
    debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
    debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
    debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
    debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
    debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
    debug2: kex_parse_kexinit: none,[email protected]
    debug2: kex_parse_kexinit: none,[email protected]
    debug2: kex_parse_kexinit: 
    debug2: kex_parse_kexinit: 
    debug2: kex_parse_kexinit: first_kex_follows 0 
    debug2: kex_parse_kexinit: reserved 0 
    debug2: mac_init: found hmac-md5
    debug1: kex: server->client aes128-ctr hmac-md5 none
    debug2: mac_init: found hmac-md5
    debug1: kex: client->server aes128-ctr hmac-md5 none
    debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
    debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
    debug2: dh_gen_key: priv key bits set: 118/256
    debug2: bits set: 484/1024
    debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
    debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
    debug3: check_host_in_hostfile: filename /home/kris/.ssh/known_hosts
    debug3: check_host_in_hostfile: match line 1
    debug3: check_host_in_hostfile: filename /home/kris/.ssh/known_hosts
    debug3: check_host_in_hostfile: match line 1
    debug1: Host '(SERVER HOST)' is known and matches the RSA host key.
    debug1: Found key in /home/kris/.ssh/known_hosts:1
    debug2: bits set: 522/1024
    debug1: ssh_rsa_verify: signature correct
    debug2: kex_derive_keys
    debug2: set_newkeys: mode 1
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug2: set_newkeys: mode 0
    debug1: SSH2_MSG_NEWKEYS received
    debug1: SSH2_MSG_SERVICE_REQUEST sent
    debug2: service_accept: ssh-userauth
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug2: key: "/home/kris/.ssh/id_rsa_kriscraig_git_CRAIGCOM-LINUX" ((nil))
    debug1: Authentications that can continue: publickey,gssapi-with-mic,password
    debug3: start over, passed a different list publickey,gssapi-with-mic,password
    debug3: preferred publickey
    debug3: authmethod_lookup publickey
    debug3: remaining preferred: 
    debug3: authmethod_is_enabled publickey
    debug1: Next authentication method: publickey
    debug1: Trying private key: "/home/kris/.ssh/id_rsa_kriscraig_git_CRAIGCOM-LINUX"
    debug3: no such identity: "/home/kris/.ssh/id_rsa_kriscraig_git_CRAIGCOM-LINUX"
    debug2: we did not send a packet, disable method
    debug1: No more authentication methods to try.
    Permission denied (publickey,gssapi-with-mic,password).

This is where I'm stuck

As you can see from the log, it's returning "no such identity" after attempting to open the private key file. I think this rejection is happening on the client-side. As far as I can tell, it's not sending anything to the server, whatsoever; just opening and abruptly closing the connection.

For the life of me, I can't figure out why it's barfing on the key files on both CentOS boxes! The permissions are set perfectly, the files are readable, neither server is using SELinux, the same keys are working perfectly fine from the Windows client, etc.

I'm good wearing the sysadmin hat; but at the end of the day, I'm a developer, not an IT person. This level of SSH debugging has taken me way outside my area of expertise. I hate to say it, but I've simply run out of ideas. According to every single thing I've been able to find on the internets, this should be working. But it's not. What am I missing?

Thanks for your help! Hopefully, one of you will recognize this issue and boot my ass in the right direction. If you need any further details, please feel free to ask.

--Kris

10
  • Oh and yes, the appropriate public key strings have been appended to authorized_keys. But like I said, I don't think it's even getting that far.
    – Kris Craig
    Apr 12, 2013 at 5:47
  • One thing I would rule out is that the server (client #2) and client #1 are talking at the right machine. You didn't mention whether you're using DNS or IPs to connect (although your exampke uses the name CRAIGCOM-LINUX).
    – tink
    Apr 12, 2013 at 6:09
  • I verified this by checking /var/log/secure on the server. It was receiving the connections every time; however, the Linux clients abruptly closed the connections immediately after opening them without actually sending any data.
    – Kris Craig
    Apr 12, 2013 at 7:05
  • 1
    @calspring Did you read the entire post? Verifying the permissions was the very first thing I tried. Also, this has nothing, whatsoever to do with Gitolite. The Gitolite credentials are not the ones that are being rejected. The credentials being rejected are the ones used to connect to the server itself. The logs I provided detail this very clearly, so I'm getting the impression you didn't actually read most of my post.
    – Kris Craig
    Apr 13, 2013 at 3:03
  • 1
    As far as hardware specs go, a good QA engineer never summarily discounts variables like that even if they're unlikely to be a factor. The best way to troubleshoot a problem like this is to use the Scientific Method, which demands that unknown variables be kept at a minimum in order to ensure the reliability of the test results. I very clearly segregated the hardware specs so that they wouldn't muffle the more pertinent data.
    – Kris Craig
    Apr 13, 2013 at 3:07

1 Answer 1

11

Please use the ssh-agent.

  • First add your key to the agent: ssh-add ~/.ssh/$keyfile.
  • Then verify that the key is loaded successfully: ssh-add -l
  • Then try to ssh to your remote box. (The agent should do the authentication.)

If there really is a problem with your key you should notice once you try to add it.

4
  • This essentially would just automate the steps that were already taken manually. The ssh-agent utility, at least as far as I can tell, wouldn't actually do anything that hasn't already been done.
    – Kris Craig
    Apr 13, 2013 at 3:10
  • The point of ssh-agent is separating the key management from the actual connection and is therefore the perfect tool for debugging your situation. Depending on the result you know where to look for the problem. - Therefore again the question: Is ssh-add able to import your key without problems?
    – michas
    Apr 13, 2013 at 5:38
  • Seems you can't traverse the ~/.ssh directory try sudo ssh -i id_rsa_kriscraig_git_CRAIGCOM-LINUX user@ip, IF this works then you need to chmod +x the ~/.ssh/* and chmod 700 ~/.ssh chmod 600~/.ssh/* May 15, 2015 at 6:20
  • Worked for a remote SSH issue in VSCode attempting to connect to a VPS. Thanks.
    – vhs
    Nov 5, 2020 at 13:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .