140

I'm seeing delays in SSH Logins. Specifically, there are 2 spots where I see a range from instantaneous to multi-second delays.

  1. Between issuing the ssh command and getting a login prompt and
  2. between entering the passphrase and having the shell load

Now, specifically I'm looking at ssh details only here. Obviously network latency, speed of the hardware and OSes involved, complex login scripts, etc can cause delays. For context I ssh to a vast multitude of linux distributions and some Solaris hosts using mostly Ubuntu, CentOS, and MacOS X as my client systems. Almost all of the time, the ssh server configuration is unchanged from the OS's default settings.

What ssh server configurations should I be interested in? Are there OS/kernel parameters that can be tuned? Login shell tricks? Etc?

2
  • are you using local accounts ? - sometimes i find pam authentication can add a delay to logging in with ssh
    – Sirex
    Jul 22, 2010 at 7:08
  • Usually local accounts. Sometimes NIS. Jul 24, 2010 at 2:39

24 Answers 24

171

Try setting UseDNS to no in /etc/sshd_config or /etc/ssh/sshd_config.

15
  • 13
    +1 that is the most common cause of delay when logging in to ssh Jul 22, 2010 at 9:22
  • 2
    "Solaris 11 note: I tried the UseDNS no setting on Solaris 11 and it corrupted the service start. Not exactly a friendly response by the service. YMMV with other *Nix variants but it seems UseDNS no may not be a valid option in Solaris 11." - comment by Keith Hoffman
    – Sathyajith Bhat
    Jan 11, 2012 at 7:21
  • 5
    I was skeptical as I use to login using the IP address (home LAN), but this solution fixed my issue. For Google's sake, though it was occurring just after, the delay had nothing to do with the "key: /home/mylogin/.ssh/id_ecdsa ((nil))" message (when running ssh -vvv). May 19, 2014 at 10:42
  • 2
    +1 for making it explicit, the file /etc/ssh/sshd_config ! I was adding in /etc/sshd_config and seeing no difference at all!!
    – Ani
    Nov 20, 2014 at 9:22
  • 3
    This small setting made my ssh login incredibly fast.. Wow.
    – BTR Naidu
    Apr 11, 2017 at 18:08
53

When I ran ssh -vvv on a server with a similar slow performance I saw a hang here:

debug1: Next authentication method: gssapi-with-mic

By editing /etc/ssh/ssh_config and commenting out that authentication method I got the login performance back to normal. Here's what I have in my /etc/ssh/ssh_config on the server:

GSSAPIAuthentication no

You can set this globally on the server, so it doesn't accept GSSAPI to authenticate. Just add GSSAPIAuthentication no to /etc/ssh/sshd_config on the server and restart the service.

3
  • I found this to be the case on with my RHEL5 servers once winbind/ad logins were configured.
    – Chad
    Feb 22, 2013 at 20:57
  • This works for me on a Ubuntu 14.04 server. Aug 3, 2016 at 14:11
  • For CentOS 7, need to set both GSSAPIAuthentication no and UseDNS no in /etc/ssh/sshd_config file.
    – Sunry
    Feb 22, 2018 at 2:34
27

For me, the culprit was IPv6 resolution, it was timing out. (Bad DNS setting at my host provider, I guess.) I discovered this by doing ssh -v, which showed which step was hanging.

The solution is to ssh with the -4 option:

ssh -4 [email protected]

6
  • 2
    I suspect more and more of us are going to see this as time passes and things (badly and) slowly accommodate IPV6. Thanks!
    – sage
    Feb 3, 2016 at 23:13
  • 6
    ... and this answer is particularly unhelpful without the debug message that confirms this is the problem.
    – E.P.
    Apr 29, 2016 at 12:21
  • It is my experience that this is a very common problem when SSH is listening on dualstack interfaces and the first thing i check when I can log in, but it takes more time time than expected.
    – Mogget
    Dec 17, 2016 at 21:44
  • Is there any chance we can fix IPv6 rather than defaulting to IPv4?
    – msrd0
    Sep 26, 2018 at 16:15
  • This is probably the reason the UseDNS no answer works. using -vvv only shows it pause at debug2: resolving "thing.net.au" port 22 no error, but this doesn't happen with -4, indicating it's a DNS IPv6 issue.
    – pmc
    Mar 7, 2019 at 23:27
21

With systemd, login may hangs on dbus communication with logind after some upgrades, then you need to restart logind

systemctl restart systemd-logind

Saw that on debian 8, arch linx, and on a suse list

5
  • 1
    Oh wow, now that was the culprit! Thanks a bunch! Mar 4, 2017 at 22:29
  • 2
    Same for me. Took a while to rule out all possible DNS and SSH issues first. Note: If the issue applies to slow sudo as well, try this first.
    – Michael
    Aug 30, 2017 at 10:36
  • I just finished some in-place upgrades from RHEL6 to RHEL7 and noticed this problem. This answer also fixed my issue.
    – user53029
    Jul 1, 2019 at 20:03
  • 1
    thank you so much, it works like a charm.
    – Bảo Nam
    Oct 14, 2019 at 12:27
  • For Devuan, my slow auth (su and ssh) problems were solved with service elogind restart as root, once I finally got there. Try sudo -i.
    – bgStack15
    Nov 29, 2020 at 21:37
10

You can always start ssh with the -v option which displays what is being done at the moment.

$ ssh -v you@host

With the information you gave I can only suggest some client side configurations:

  • Since you write that you are entering passwords manually, I would suggest that you use public key authentification if possible. This removes you as a speed bottleneck.

  • You could also disable X-forwarding with -x and authentication forwarding with -a (these might already be disabled by default). Especially disabling X-forwarding can give you a big speed improvement if your client needs to start an X-server for the ssh command (e.g. under OS X).

Everything else really depends on what kinds of delays you experience where and when.

1
  • Good hint about verbosity, you can also increase it by having more v's. Up to 3 IIRC.
    – vtest
    Sep 22, 2010 at 18:51
9

Regarding the 2. point, here is an answer that don't require to modify the server nor require to have root/administrative privileges.

You need to edit your "user ssh_config" file which is:

vi $HOME/.ssh/config

(Note: you would have to create the directory $HOME/.ssh if it does not exist)

And add:

Host *
  GSSAPIAuthentication no
  GSSAPIDelegateCredentials yes

You can do so on a per host basis if required :) example:

Host linux-srv
  HostName 192.158.1.1
  GSSAPIAuthentication no
  GSSAPIDelegateCredentials yes

Make sure the IP address match your server IP. One cool advantage is that now ssh will provide autocomplete for this server. So you can type ssh lin + Tab and it should autocomplete to ssh linux-srv.

7

I found that PAM was reading the file /var/log/btmp, which had become huge as a result of people trying to brute-force my server. This was leading to login times of a minute. Clearing this file solved the problem.

3
  • 1
    Right, this one hit me as well. Check out the following PAM issue: github.com/linux-pam/linux-pam/issues/270 Aug 22, 2021 at 14:48
  • 1
    I experienced this, too. Logrotate is supposed to clean up /var/log/btmp, but if logrotate somehow gets disabled, then the file can end up growing larger and larger.
    – amcnabb
    Dec 13, 2021 at 20:25
  • same error, We have been attacked by Redis virus, so remove that files works, I also remove all ssh key, uninstall ssh and re-install it. Jan 24 at 6:52
6

I found that restarting systemd-logind.service only cured the problem for a few hours. Changing UsePAM from yes to no in sshd_config has resulted in fast logins, although motd is no longer displayed. Comments about security issues?

3
  • I had gone through EVERY other suggestion here, and this is the only thing which fixed the issue on my Samba4 enable server... THANKS! Oct 21, 2016 at 12:35
  • WARNING: 'UsePAM no' is not supported in Red Hat Enterprise Linux and may cause several problems. Jan 3, 2017 at 11:55
  • This "fixes" ssh but "su" still has 25 sec PAM auth delay.
    – JohnMudd
    Dec 15, 2022 at 16:24
4

Check /etc/resolv.conf on the server to be sure that DNS server, listed in this file, works ok, and delete any non-working DNS.

Sometimes it is very helpful.

4

Note: This started as a "How to debug", tutorial, but ended up being the solution that helped me on an Ubuntu 16.04 LTS server.

TLDR: Run landscape-sysinfo and check if that command takes a long time to finish; it's the system information printout on a new SSH login. If that's the case, you may be able to disable the execution of /etc/update-motd.d/50-landscape-sysinfo, which does call that command, by chmod -x-ing it. Note that this command isn't available on all systems, the landscape-common package installs it.


Start a second ssh server on another port on the machine that has the problem, do so in debug mode, which won't make it fork and will print out debug messages:

sudo /usr/sbin/sshd -ddd -p 44321

connect to that server from another machine in verbose mode:

ssh -vvv -p 44321 username@server

My client outputs the following lines right before starting to sleep:

debug1: Entering interactive session.
debug1: pledge: network

Googling that isn't really helpful, but the server logs are better:

debug3: mm_send_keystate: Finished sending state [preauth]
debug1: monitor_read_log: child log fd closed
debug1: PAM: establishing credentials
debug3: PAM: opening session
---- Pauses here ----
debug3: PAM: sshpam_store_conv called with 1 messages
User child is on pid 28051

I noticed that when I change UsePAM yes to UsePAM no then this issue is resolved.

Not related to UseDNS or any other setting, only UsePAM affects this problem on my system.

I have no clue why, and I'm also not leaving UsePAM at no, because I do not know which the side-effects are, but this lets me continue investigating.

So please don't consider this to be an answer, but a first step to start finding out what's wrong.


So I continued investigating, and ran sshd with strace (sudo strace /usr/sbin/sshd -ddd -p 44321). This yielded the following:

sendto(4, "<87>Nov 20 20:35:21 sshd[2234]: "..., 110, MSG_NOSIGNAL, NULL, 0) = 110
close(5)                                = 0
stat("/etc/update-motd.d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
umask(022)                              = 02
rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7f15dce784b0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7f15dce784b0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7ffde6152d2c) = 2385
wait4(2385, # BLOCKS RIGHT HERE, BEFORE THE REST IS PRINTED OUT # [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2385

The line /etc/update-motd.d made me suspicious, apparently the process waits for the result of the stuff that is in /etc/update-motd.d

So I cd'd into /etc/update-motd.d and ran a sudo chmod -x * in order to inhibit PAM to run all the files which generate this dynamic Message Of The Day, which includes system load and if packages need to be upgraded, and this solved the issue.

This is a server based on an "energy-efficient" N3150 CPU which has a lot of work to do 24/7, so I think that collecting all this motd-data was just too much for it.

I may start to enable scripts in that folder selectively, to see which are less harmful, but specially calling landscape-sysinfo is very slow, and 50-landscape-sysinfo does call that command. I think that is the one which causes the biggest delay.

After reenabling most of the files I came to the conclusion that 50-landscape-sysinfo and 99-esm were the cause for my troubles. 50-landscape-sysinfo took about 5 seconds to execute and 99-esm about 3 seconds. All the remaining files about 2 seconds altogether.

Neither 50-landscape-sysinfo and 99-esm are crucial. 50-landscape-sysinfo prints out interesting system stats (and also if you're low on space!), and 99-esm prints out messages related to Ubuntu Extended Security Maintenance

Finally you can create a script with echo '/usr/bin/landscape-sysinfo' > info.sh && chmod +x info.sh and get that printout upon request.

2

Besides the DNS issues already mentioned, if you're ssh'ing into a server with many NFS mounts, then there can be a delay between password and prompt as the quota command checks for your usage/quota on all filesystems not mounted with the noquota. On Solaris systems, you can see this in the default /etc/profile and skip it by running touch $HOME/.hushlogin .

2

This is probably only specific to the Debian/Ubuntu OpenSSH, which includes the user-group-modes.patch written by one of the Debian package maintainers. This patch allows the ~/.ssh files to have the group writable bit set (g+w) if there is only one user with the same gid as that of the file. The patch's secure_permissions() function does this check. One of the phases of the check is to go through each passwd entry using getpwent() and compare the gid of the entry with the gid of the file.

On a system with many entries and/or slow NIS/LDAP authentication, this check will be slow. nscd does not cache getpwent() calls, so every passwd entry will be read over the network if the server is not local. On the system I found this, it added about 4 seconds for each invocation of ssh or login into the system.

The fix is to remove the writable bit on all of the files in ~/.ssh by doing chmod g-w ~/.ssh/*.

1
  • Seet! Thanks a lot. I searched for this a long time. God catch!
    – mofoe
    Aug 24, 2022 at 13:40
1

Work fine.

# uname -a
SunOS oi-san-01 5.11 oi_151a3 i86pc i386 i86pc Solaris
# ssh -V
Sun_SSH_1.5, SSH protocols 1.5/2.0, OpenSSL 0x009080ff
# echo "GSSAPIAuthentication no" >> /etc/ssh/sshd_config
# echo "LookupClientHostnames no" >> /etc/ssh/sshd_config
# svcadm restart ssh

UseDNS no do not work with OpenIndiana !!!

Read "man sshd_config" for all the options

"LookupClientHostnames no" if your server can not resolve

1

If none of the above answers works and you're facing dns reverse lookup problems you can also check if nscd (name service cache daemon) is installed and running.

If this is the problem it is because you have no dns cache, and each time you query for a hostname that is not on your hostfile you send the question to your name server instead of looking in your cache

I tried all the above options and the only change that worked was start nscd.

You should also verify the order to make dns query resolution in /etc/nsswitch.conf to use hosts file first.

1

To complete all the answers showing that DNS resolutions can slow your ssh login, sometimes, a firewall rules is missing. For example, if you DROP all the INPUT paquets by default

iptables -t filter -P INPUT DROP

then you'll have to accept INPUT for ssh port and DNS request

iptables -t filter -A INPUT -p tcp --dport 53 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 53 -j ACCEPT
1

ssh -vvv connection went really fine until it hung on the system trying to get the terminal for at least 20 Seconds:

debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting [email protected]
debug1: Entering interactive session.
... waiting ... waiting ... waiting

After doing a systemctl restart systemd-logind on the server I had instant connection again!

This was on debian8! So systemd was the issue here!

Note: Bastien Durel already gave an answer for this issue, however it lacks the debug information. I hope this is helpful to someone.

1
  • I had the same issue with "Entering interactive session" hanging on RHEL7 (CentOS 7) and solved it by commenting out session [default=1] pam_lastlog.so nowtmp showfailed in /etc/pam.d/postlogin. Apparently updating the lastlog file was incredibly slow on my container-based OpenVZ VPS. Dec 28, 2018 at 4:32
1

I have recently found another cause of slow ssh logins.

Even if you have UseDNS no in /etc/sshd_config, sshd may still perform reverse DNS lookups if /etc/hosts.deny has an entry like:

nnn-nnn-nnn-nnn.rev.some.domain.com

That might happen if you have DenyHosts installed in your system.

It would be great if someone knew how to make DenyHosts avoid putting this kind of entry in /etc/hosts.deny.

Here is a link, to the DenyHosts FAQ, on how to remove entries from /etc/hosts.deny - see How can I remove an IP address that DenyHosts blocked?

1

We may find that the preferred name resolution method isn't the host file and then DNS.

For example, this would be the usual configuration:

[root@LINUX1 ~]# cat /etc/nsswitch.conf|grep hosts
#hosts:     db files nisplus nis dns
hosts:      files dns myhostname

First, hosts file is reached (option: files) and then DNS (option: dns), however we can find that another name resolution system has been added that is not operational and is causing us the slowness in trying to do the reverse resolution.

If the name resolution order isn't correct, you can change it at: /etc/nsswitch.conf

Extracted from: http://www.sysadmit.com/2017/07/linux-ssh-login-lento.html

1

I tried all the answers but none of them worked. finally I find out my problem:

first I run sudo tail -f /var/log/auth.log so I can see the log of ssh then in another session run ssh 172.16.111.166 and noticed waiting on

/usr/bin/sss_ssh_knownhostsproxy -p 22 172.16.111.166

after searching I located this line in /etc/ssd/ssh_config

ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

I commented it and the delay gone

1

This thread is already providing a bunch of solutions but mine is not given here =). So here it is. My problem (took about 1 minutes to ssh login into my raspberry pi), was due to a corrupted .bash_history file. Since the file is read at login, this was causing the login delay. Once I removed the file, login time went back to normal, like instantaneous.

Hope this will help some other people.

0

For me I needed GSSAPI, and I didn't want to turn off reverse DNS lookups. That just didn't seem like a good idea, so I checked out the main page for resolv.conf. It turns out that a firewall between me and the servers I was SSHing to, was interfering with DNS requests, because they weren't in a form that the firewall expected. In the end, all I needed to do was add this line to resolv.conf on the servers that I was SSHing to:

options single-request-reopen

0

Remarkably, a package update of bind on CentOS 7 broke named, now stating in the log that /etc/named.conf had a permissions problem. It had worked well for months with 0640. Now it wants 0644. This makes sense as the named daemon belongs to the 'named' user.

With named down everything was slow, from ssh logins to page serving from the local web server, sluggish LAMP apps etc, most probably because every request would time out on the dead local server before looking up to an external, secondary DNS that is configured.

0

For me there was an issue in my local /etc/hosts file. So ssh was trying two different IP (one wrong) which took forever to time-out.

Using ssh -v did the trick here:

$ ssh -vvv remotesrv
OpenSSH_6.7p1 Debian-5, OpenSSL 1.0.1k 8 Jan 2015
debug1: Reading configuration data /home/mathieu/.ssh/config
debug1: /home/mathieu/.ssh/config line 60: Applying options for remotesrv
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to remotesrv [192.168.0.10] port 22.
debug1: connect to address 192.168.0.10 port 22: Connection timed out
debug1: Connecting to remotesrv [192.168.0.26] port 22.
debug1: Connection established.
1
  • The /etc/hosts of the server?
    – Daniel F
    Nov 20, 2018 at 19:17
0

If ssh taking time then we can follow the steps : There are 2 files

  1. ssh_config
  2. sshd_config --> will change in sshd_config

We can uncomment the line no 115

[root@mycentos ~]# vi /etc/ssh/sshd_config

UseDNS no -> default #UseDNS no --> remove # -> UseDNS no only

Save the sshd file and restart the sshd service

[root@mycentos ~]# systemctl restart sshd

If not,again take time then we can follow below change more : (Uncomment below lines )

  1. GatewayPorts no

  2. PermitTTY yes

  3. UseDNS no -> line no 115

  4. GSSAPIAuthentication no -> line no 79

Restart the sshd service and try to take the ssh

[root@mycentos ~]# systemctl restart sshd

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .