3

I give up:

I have a cronjob script that Ive been running for years (out of a RHEL 6.7 machine) to remote into other RHEL machines via ssh. Works flawlessly against RHEL5 and 6 but fails against RHEL7. At its simplest form the script breaks down like this:

ssh -tttvi /home/robot/.ssh/passwdlesskey  robot@${ThatIP} sudo -l

Across all remote machines (RHEL6 & 7), the respective sudoers file contains

Defaults    requiretty

As expected, the script executes well from commandline against RHEL7. Again, from cron it works against RHEL 5 & 6 but fails against 7:

debug1: read_passphrase: can't open /dev/tty: No such device or address
Host key verification failed.

One peculiar thing to note is the difference in the SSH outputs when running commandline vs cron against a RHEL7 server

              C  R  O  N                                                                 C O M M A N D L I N E
------------------------------------------------------------------------+------------------------------------------------------------------------
debug1: skipped DNS lookup for numerical hostname                       | debug1: skipped DNS lookup for numerical hostname
WARNING: ECDSA key found for host 10.96.16.108                          | debug1: Host '10.96.16.108' is known and matches the ECDSA host key.
in /home/robot/.ssh/known_hosts:416                                     | debug1: Found key in /home/robot/.ssh/known_hosts:416
ECDSA key fingerprint ee:67:90:d3:c3:b8:db:c7:d3:6a:68:6a:78:fd:25:da.  | debug1: SSH2_MSG_NEWKEYS sent
+--[ECDSA  256]---+                                                     | debug1: expecting SSH2_MSG_NEWKEYS
|                 |                                                     | debug1: SSH2_MSG_NEWKEYS received
|                 |                                                     | debug1: Roaming not allowed by server
|                 |                                                     | debug1: SSH2_MSG_SERVICE_REQUEST sent
|                 |                                                     | debug1: SSH2_MSG_SERVICE_ACCEPT received
|        S=       |                                                     | debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password                                          
|       .= +      |                                                     | debug1: Next authentication method: publickey
|       ..= +...  |                                                     | debug1: Offering DSA public key: /home/robot/.ssh/passwdlesskey
|      ..+.*o=o.  |                                                     | debug1: Server accepts key: pkalg ssh-dss blen 433
|       o+*o+Eo   |                                                     | debug1: Authentication succeeded (publickey).
+-----------------+                                                     | Authenticated to 10.96.16.108 ([10.96.16.108]:22).
                                                                        | debug1: channel 0: new [client-session]
debug1: read_passphrase: can't open /dev/tty: No such device or address | debug1: Entering interactive session.
Host key verification failed.                                           | debug1: Sending command: sudo -l
------------------------------------------------------------------------+------------------------------------------------------------------------

One theory I had: similar to the way scp/sftp does not like interactive shell output, I figured the randomart displayed by the RHEL7 server is interfering with my script. But even replacing ssh -v with -q does not help.

sshd's LogLevel is DEBUG2

Mar  9 10:51:01 rhel7test sshd[26198]: debug1: Forked child 26209.
Mar  9 10:51:01 rhel7test sshd[26209]: Set /proc/self/oom_score_adj to 0
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: inetd sockets after dupping: 3, 3
Mar  9 10:51:01 rhel7test sshd[26209]: Connection from 10.96.16.148 port 55171 on 10.96.16.108 port 22
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: Client protocol version 2.0; client software version OpenSSH_5.3
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: Enabling compatibility mode for protocol 2.0
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: Local version string SSH-2.0-OpenSSH_6.6.1
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: fd 3 setting O_NONBLOCK
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: Network child is on pid 26210
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: SELinux support enabled [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: permanently_set_uid: 74/74 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: list_hostkey_types: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: SSH2_MSG_KEXINIT received [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: none,zlib@openssh.com [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: none,zlib@openssh.com [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit:  [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit:  [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: first_kex_follows 0  [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: reserved 0  [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit:  [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit:  [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: first_kex_follows 0  [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_parse_kexinit: reserved 0  [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: mac_setup: setup hmac-md5 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: kex: client->server aes128-ctr hmac-md5 none [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: mac_setup: setup hmac-md5 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: kex: server->client aes128-ctr hmac-md5 none [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: monitor_read: 0 used once, disabling now
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: bits set: 504/1024 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: bits set: 532/1024 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: monitor_read: 6 used once, disabling now
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: kex_derive_keys [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug2: set_newkeys: mode 1 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: Connection closed by 10.96.16.148 [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: do_cleanup [preauth]
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: monitor_read_log: child log fd closed
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: do_cleanup
Mar  9 10:51:01 rhel7test sshd[26209]: debug1: Killing privsep child 26210

Reading similar posts, Ive already

  • checked the permissions on /dev/tty
  • made sure the password-less keys are correct
  • made sure there's no conflicting entries in known_hosts file
  • checked environment variables for anything strange
  • kicked up debugging to level 3 on executing side (-vvv) with no new information to glean from the output.

Finally, because Im sure it will be asked, the sshd_config of RHEL6 & 7

         R H E L 7                                                                 R H E L 6
--------------------------------------------------------------------------------+---------------------------------------------------------------------
HostKey                         /etc/ssh/ssh_host_rsa_key                       | 
HostKey                         /etc/ssh/ssh_host_ecdsa_key                     |
HostKey                         /etc/ssh/ssh_host_ed25519_key                   |
SyslogFacility                  AUTHPRIV                                        | AUTHPRIV
LogLevel                        DEBUG2                                          | 
PermitRootLogin                 no                                              | no
StrictModes                     yes                                             |
MaxAuthTries                    3                                               |
MaxSessions                     10                                              |
PubkeyAuthentication            yes                                             |
AuthorizedKeysFile              .ssh/authorized_keys                            |
IgnoreRhosts                    yes                                             |
PermitEmptyPasswords            no                                              |
PasswordAuthentication          yes                                             | yes
ChallengeResponseAuthentication no                                              | no
GSSAPIAuthentication            yes                                             | yes
GSSAPICleanupCredentials        no                                              | yes
UsePAM                          yes                                             | yes
X11Forwarding                   yes                                             | yes
PrintMotd                       no                                              |
PrintLastLog                    no                                              |
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES   | LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT               | LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE                                     | LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS                                                            | XMODIFIERS
Subsystem               sftp  /usr/libexec/openssh/sftp-server                  | sftp  /usr/libexec/openssh/sftp-server

So Im at a loss as to what the cause of the problem is.

  • Can you try running your script with "ssh -o StrictHostKeyChecking=no " and see if that resolves the issue? – VDR Mar 9 '16 at 18:46
  • VDR: I would not have thought that particular option would have solved the issue, but it has. Changing to ssh -ttttvi /home/robot/.ssh/passwdlesskey -o StrictHostKeyChecking=no robot@${ThatIP} sudo -l did exactly what it should. Now I only need to test against the other servers to make sure my script doesnt break there (or if I have to branch off more code explicitly for RHEL7 servers) – Ogal Finklestein Mar 9 '16 at 19:16
2

It is not sudo asking for TTY, but the ssh client, which needs from you to do the only security thing about ssh, that you have to do before establishing secure connection. It is verification of hostkey, that requires manual input from user, if it is not configured in advance. It is not related to the system version, but to the configuration.

You need to store the servers host key (public part) to the client's known_hosts. There are several possibilities how to do that:

  • As the user (robot) who is running the cron (sudo -i -u robot), try to connect to the remote server:

    ssh -i /home/robot/.ssh/passwdlesskey  robot@${ThatIP}
    

    and manual write "yes" to the prompt

  • Use ssh-keyscan tool, which does exactly the same thing, but does not have to run interactive (make sure if you run it as different user, that the target user will have correct access to that file):

    ssh-keyscan {ThatIP} >> /home/robot/.ssh/known_hosts
    
  • Thanks for your answer but I am not new to all of this to not have already been through all those steps. And yes I am quite aware that it's ssh and not sudo, but I added the sudo information to prevent people from commenting on it. – Ogal Finklestein Mar 9 '16 at 19:15
0

As user VDR commented I need to add

-o StrictHostKeyChecking=no

as an additional option when calling ssh through cron.

  • It is not ideal solution. What is the client side system where you run the code (this is obviously problem)? – Jakuje Mar 9 '16 at 19:27
  • Jakuje, how can you assume this? I know the code, and Ive posted the code snippet in my OP. I took the recommendation from VDR, implemented it and it worked. Problem solved; solution found. – Ogal Finklestein Mar 9 '16 at 19:30
0

Same scenario--I updated the target system from CentOS 6 to CentOS 7, and suddenly my cron job would get Host key validation failed errors, but if I ran it from command line, it would be fine. I was totally unaware of the ssh-keyscan tool, so thanks to @Jakuje for writing about that. In my case, running the ssh-keyscan tool output absolutely nothing, so I added -v, and got:

>ssh-keyscan -v hostname.example.com
debug1: match: OpenSSH_6.6.1 pat OpenSSH*
debug1: hostname.example.com doesn't support ssh1

Eish. Looks like SSH on the source machine doesn't support SSH2. So the I looked into the version of ssh on the source system:

>sudo yum list installed | grep ssh
openssh.x86_64                           4.3p2-82.el5                  installed
openssh-askpass.x86_64                   4.3p2-82.el5                  installed
openssh-clients.x86_64                   4.3p2-82.el5                  installed
openssh-server.x86_64                    4.3p2-82.el5                  installed

Nice. Only several years behind there. This post http://www.jbytechnologies.com/linux/sshd-add-dsa-key.html helped me dig deeper to note that CentOS 7 disables DSA keys. Solution? Generate and enable the DSA key on CentOS 7. But, DSA keys are weaker than RSA, so that would be unhardening my new server.

Optimal solution in my case: upgrade openssh on the source machine.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.