1

The server in question is a runner and builds projects with a system user gitlab-runner. That user is used to SSH into other servers to deploy code, etc..

Always worked fine until we added a new server as destination. The command SSH now always fails with the error 'Host key verification failed.'. The error is also thrown when I try other servers that worked before. The known_hosts file is cleared but SSH doesn't ask to add the server to known_hosts anymore, it returns the error message directly.

I checked the permissions of the ~/.ssh folder and files. These are correct (.ssh: 700, known_hosts: 600, id_rsa: 600, id_rsa.pub: 644). Als rebooted the server but no success.

It feels like SSH is not working correctly. Here is a debug output of connecting to a server through SSH.

OpenSSH_7.2p2 Ubuntu-4ubuntu2.4, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to megatron.domain.com [10.139.20.204] port 22.
debug1: Connection established.
debug1: identity file /home/gitlab-runner/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to megatron.achillescm.nl:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:/zgPQuuy6sG8UuLG9EHFSFAuY1QYNvQzKSyNYq//DJ0
debug1: read_passphrase: can't open /dev/tty: No such device or address
Host key verification failed.

Anybody an idea?

  • ssh is indeed trying to ask you about accepting (and adding) the host key, but for some reason /dev/tty cannot be opened. I've never seen that, and it suggests something is quite messed up on your system, but with the info given I have no clue what. (The function name read_passphrase is somewhat misleading here; it is also called from confirm in ssh_connect.c) If you manually rebuild known_hosts e.g. with ssh-keyscan it should avoid this particular problem, although what else may go wrong I cannot say. – dave_thompson_085 Feb 20 '18 at 5:19
  • @dave_thompson_085 Thx for the comment. The system is a very simple setup; ubuntu + gitlab runner, nothing more. – Sven van Zoelen Feb 21 '18 at 11:04
1

I found the problem. The problem was that the known_hosts file had the wrong permissions.

It was set to 600 and it should be 644.

  • It gives me error: Permissions 0644 for 'SSH_PRIVATE_KEY' are too open. It is required that your private key files are NOT accessible by others. – Sky Nov 8 at 13:08
0
debug1: identity file /home/gitlab-runner/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory

Above lines suggest that .ssh/id_rsa exists (type 1), but its public key (.ssh/id_rsa.pub) for the user doesn't or got some problems (type -1).

So make sure the file ~/.ssh/id_rsa exists, has 600 permission, is owned by the user. And the ~/.ssh/id_rsa.pub matches/belongs to the same identity (as per this post).

Here is a simple test to run as a sudo access user (e.g. root):

sudo -u gitlab-runner sh -c 'cd; wc -l .ssh/id_rsa; stat .ssh/id_rsa; head -n1 .ssh/id_rsa'

See also: What does "key_load_public: no such file or directory" mean?

  • Exactly the opposite; type 1 means id_rsa could be read and does contain an RSA key. (Other algorithms are other positive numbers; negative one is an error.) corrected: It isn't finding the subsequent file id_rsa-cert. – dave_thompson_085 Feb 20 '18 at 5:23
  • The command you specified returns bash: line 0: cd: /root: Permission denied wc: .ssh/id_rsa: Permission denied stat: cannot stat '.ssh/id_rsa': Permission denied. When I execute the command after su gitlab-runner - then it returns: File: '.ssh/id_rsa' Size: 3243 Blocks: 8 IO Block: 4096 regular file Device: fd01h/64769d Inode: 283738 Links: 1 Access: (0600/-rw-------) Uid: ( 999/gitlab-runner) Gid: ( 998/gitlab-runner) Acces.. – Sven van Zoelen Feb 20 '18 at 7:33
0

Sometimes, the same problem arises if /dev/tty permissions are too restrictive. Then you can only ssh to a server properly as user root, nobody else can. The fix is to chmod 777 /dev/tty so the others can do it.

  • 666 is probably just as good (or maybe better). I don't know of any reason to give execute permission to a device. – Scott May 27 '18 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.