ssh timeout option taking longer than specified

Question

Trying to run a script which confirms the date setting on a large number of servers on local network. The servers are all local and if healthy, I would expect a < 1 second response time. To make the script run at useful speeds, I want to connection timeout to be as low as possible.

So I try this as my main test in a loop. (The IP address would be variable)

ssh -v -o ConnectTimeout=1 -o ConnectionAttempts=1 10.x.x.x date

For healthy servers this runs fine, but I'm seeing some boxes that hang and appear to ignore the Timeout values. Hanging 1 to 2 minutes rather than the 1 second I would expect. Verbose output looks like this.

OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /root/.ssh/config
debug1: Applying options for *10.x.x.x*
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to *10.x.x.x* [*10.x.x.x*] port 22.
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/aws_rsa type -1
debug1: identity file /root/.ssh/aws_rsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

(here is where is sits for one to two minutes)

Connection to *10.x.x.x* timed out while waiting to read

One detail I'm noticing is that there is 'some' sort of connection, that's occurring, but it a failed connection. The server is not healthy and not responding in the expected way. Is there another option I can choose that would detect that and fail more quickly when there a protocol issue?

Answer 1

From manual page for ssh_config:

ConnectTimeout

Specifies the timeout (in seconds) used when connecting to the SSH server, instead of using the default system TCP timeout. This value is used only when the target is down or really unreachable, not when it refuses the connection.

It might not be absolutely clear from these lines, but the timeout is used only for connect() system call, basically before line

debug1: Connection established.

and not for every other answer from the server. For this, there is option ServerAliveInterval, which pings server after every X seconds to make sure the server is responding after connection (by default turned off). But you are hitting the case when your TCPKeepAlive times out, which is yet another chapter.