0

I'm running into SSH errors when connecting to a server as a particular user when using key based authentication with an associated password on the key.

This is on an IBM i system.

I've added the public key to the ~/.ssh/authorized_keys file on the server for that user. Their home directory, along with the ~/.ssh directory and the /.ssh/authorized_keys file do not have write permissions for anyone other than the owner. And the owner is properly set for all of them. Interesting enough, if I move those same keys to another user, the password is accepted. Alternatively, if I use a key pair which works for a different user for the user JAV, I get the same behavior. So it seems there is something about the user "JAV" on the server-side which is at the root of this.

What's odd is that the connection attempt doesn't fail until after the user is prompted for a password. Everything I can find points to file permissions being inaccurate, but I've matched the permissions for relevant files between the user which works and the user which doesn't - and their identical.

JAV is the user which doesn't work.

Permissions for /home/JAV and .ssh:

bash-4.4$ pwd
/home/JAV
bash-4.4$ ls -la
total 300
drwx--S---   4 jav  0  24576 Oct 17 14:41 .
drwxrwsrwx 102 qsys 0  61440 Oct 17 15:21 ..
-rw-------   1 jav  0   2108 Oct 17 15:32 .bash_history
-rwx------   1 jav  0    204 Dec 18  2018 .bash_profile
-rwx------   1 jav  0     83 May  5  2017 .bashrc
drwxrwsrwx   3 jav  0   8192 May  9  2018 .eclipse
-rwx------   1 jav  0    559 Oct 17 16:45 .profile
-rw-------   1 jav  0    546 Oct 22  2018 .sh_history
drwx--S---   2 jav  0   8192 Oct 17 16:55 .ssh
bash-4.4$

Very very verbose output when trying a failed attempt:

┌─[dl @ PC-1802]─[~/.ssh]─[575]
└─[$]› ssh -vvv JAVDL
OpenSSH_7.6p1, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /c/Users/dl/.ssh/config
debug1: /c/Users/dl/.ssh/config line 55: Applying options for JAVDL
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "mydomain.local" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to mydomain.local [10.0.15.8] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dl/.ssh/dl_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dl/.ssh/dl_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to mydomain.local:22 as 'jav'
debug3: hostkeys_foreach: reading file "/c/Users/dl/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /c/Users/dl/.ssh/known_hosts:1
debug3: record_hostkey: found key type RSA in file /c/Users/dl/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 2 keys from mydomain.local
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes256-cbc,aes192-cbc
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes256-cbc,aes192-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64-etm@openssh.com compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:7Z25Q7mljYwlFOHC0SPGiCraFwbW/Pm4ecbfLL/w29E
debug3: hostkeys_foreach: reading file "/c/Users/dl/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /c/Users/dl/.ssh/known_hosts:1
debug3: record_hostkey: found key type RSA in file /c/Users/dl/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 2 keys from mydomain.local
debug3: hostkeys_foreach: reading file "/c/Users/dl/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /c/Users/dl/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 10.0.15.8
debug1: Host 'mydomain.local' is known and matches the ECDSA host key.
debug1: Found key in /c/Users/dl/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /c/Users/dl/.ssh/dl_rsa (0x0), explicit
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,keyboard-interactive
debug3: start over, passed a different list publickey,keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /c/Users/dl/.ssh/dl_rsa


Enter passphrase for key '/c/Users/dl/.ssh/dl_rsa':


debug3: sign_and_send_pubkey: RSA SHA256:YVE0kS2H4/n5umGS4Gv0J+aWFljM4pauXG8KEe1+2YI
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,keyboard-interactive
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,keyboard-interactive
debug3: userauth_kbdint: disable: no info_req_seen
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
jav@mydomain.local: Permission denied (publickey,keyboard-interactive).

┌─[dl @ PC-1802]─[~/.ssh]─[576]
└─[$]›

Permission on ~/.ssh/authorized_keys:

bash-4.4$ pwd
/home/JAV/.ssh
bash-4.4$ ls -la
total 40
drwx--S--- 2 jav 0  8192 Oct 17 17:22 .
drwx--S--- 4 jav 0 24576 Oct 17 14:41 ..
-rwx------ 1 jav 0  1303 Oct 17 16:55 authorized_keys
bash-4.4$

SSH host alias definition I'm using:

Host JAVDL
  HostName mydomain.local
  Port 22
  User jav
  IdentityFile ~/.ssh/dl_rsa

I'd also like to note that in our sshd_config file we have explicit deny/allow directives.

Portion of sshd_config

DenyUsers root
AllowUsers dl jav

I can't find any indication of what debug3: receive packet: type 51 means. Maybe my Googling skills are lacking, but I can't even find a reference to what the various packet types are for openssh, not even in the RFC.

  • Do you have verbose output of the server-side sshd logs? (That's where the check and rejection are happening, after all.) – user1686 Oct 18 '19 at 5:04
0

I can't find any indication of what debug3: receive packet: type 51 means. Maybe my Googling skills are lacking, but I can't even find a reference to what the various packet types are for openssh, not even in the RFC.

Packet 51 means SSH_MSG_USERAUTH_FAILURE as described in the RFC 4252:

https://tools.ietf.org/html/rfc4252#section-6

This means the key was rejected for some reason, which is recorded in the server logs (or will be recorded if you increase logLevel on the server).

0

First, SSHD logging isn't enabled by default on IBM i. To set it up, follow this article: https://www.ibm.com/support/pages/sshd-can-use-unix-syslog-facilities-logging

Then, after increasing the logLevel on the server to DEBUG3 I found the entry:

sshd[262719]: User jav not allowed because shell /QOpenSys/QIBM/ProdData/OPS/tools/bin/bash does not exist

That file does not exist on our system, but it also shouldn't exist as we installed bash from IBM's YUM repository. That is the location of bash if using 5733OPS to obtain bash, which is now considered obsolete.

After some searching I determined there is a UDF which sets the default PASE shell for a given user. QSYS2.SET_PASE_SHELL_INFO() as explained here: http://archive.ibmsystemsmag.com/blogs/open-your-i/august-2017/tech-tip-be-like-a-turtle/. After querying the value for the user "jav" and comparing it to a user for which this works, I found the user "jav" had the value set to /QOpenSys/QIBM/ProdData/OPS/tools/bin/bash while the value for all other users was NULL. Sure enough, after setting the value to NULL, everything worked as expected.

I gather you can set the default shell using this method instead of using startup files such as .profile. We just use startup files though.

Query to check the value for all users:

SELECT AUTHORIZATION_NAME, PASE_SHELL_PATH
FROM QSYS2.USER_INFO;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.