Still getting “ssh failure: Offending RSA key in /var/lib/sss/pubconf/known_hosts” despite removing the specified key from “known_hosts”

Question

I know similar questions have been asked many times, but there's a difference here:

Trying to ssh to the server I get:

$ ssh cl11lx
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
07:51:03:4e:5e:ba:e8:44:70:77:cb:57:78:57:59:35.
Please contact your system administrator.
Add correct host key in /nethome/ajalali/.ssh/known_hosts to get rid of this message.
Offending RSA key in /var/lib/sss/pubconf/known_hosts:8
  remove with: ssh-keygen -f "/var/lib/sss/pubconf/known_hosts" -R cl11lx
ECDSA host key for cl11lx has changed and you have requested strict checking.
Host key verification failed.

The most relevant question I found is here, which is closed as a duplicate of this one which itself is closed as off-topic. But they're not the same anyway. Another related question is asked here.

In most cases, as we know, you can fix the problem by replacing the server key, for example by running:

ssh-keygen -R <host>

But neither this solution, nor removing the offending key from the file /var/lib/sss/pubconf/known_hosts did solve my problem.

As soon as I remove that whole file, or remove the offending key, the key is back.

Please note that all my systems are members of a freeipa server, which itself is an LDAP/Kerberos on the backend.

Answer 1

As seen in the posted log, the offending key is not in its normal place, i.e. $HOME/.ssh/known_hosts, but it's rather located in /var/lib/sss/pubconf/known_hosts. This suggests that System Security Services Daemon aka sssd is fetching the offending key from another source.

This is why removing the key from the file has no effect, because it's being re-fetched from a server over and over again.

In this case, the server is a FreeIPA server. Therefore to fix the problem, the offending key sent by the IPA server has to be changed.

This can be done by manually changing the key on the server, or removing the host information from the server completely, and installing the client (via ipa-client-install) again.

My problem happened after re-installing the OS on the client. Therefore it is a sensible option for me to remove the host info from the server and install the IPA client again.

Answer 2

TL;DR: On the server you want to connect to, run:

ipa host-mod --sshpubkey="`cat /etc/ssh/ssh_host_ed25519_key.pub`" --sshpubkey="`cat /etc/ssh/ssh_host_ecdsa_key.pub`" --sshpubkey="`cat /etc/ssh/ssh_host_rsa_key.pub`"  `hostname`

What's happening here is that the ssh client is retrieving the server host's public key from the IPA Server, but the IPA Server doesn't know that the host's public key has changed.

ssh gets the old key, and ssh becomes unhappy.

To update the keys in the IPA server, use ipa host-mod or the IPA WebGUI.

If you only need ed25519, it's fine to skip the other keys. E.g:

ipa host-mod --sshpubkey="`cat /etc/ssh/ssh_host_ed25519_key.pub`" `hostname`

Using the WebGUI requires more cut and paste to add keys, so depending on what you want to do, the command line may be the easier option.

If you want to view or modify ssh keys using the WebGUI:

1. navigate to Hosts->'your host'->Settings
2. scroll down to SSH Keys
3. view or update using the show/set, delete and add buttons
4. scroll up and save