Updates
- I've narrowed it down to only occurring on the wireless network at work. From home, I experience no lag/delay
- It also seems that after "heavy" SSH use and tunneling through one or two machines, the network drops me completely and I am forced to manually reconnect
- Looking at WireShark captures of the SSH session from OpenSSH versus JSch, it seems that OpenSSH has a handful of
TCP Dup Ack,TCP Retransmission, andTCP Spurious Retransmissionpackets. But these should sort themselves out, being TCP..
Problem
Recently when connecting to remote machines using SSH, I've noticed that there is a distinct delay of around 1 second between me typing something and the characters actually appearing. If I hold down keys, they'll appear in "batches" of around 20 characters at ~1s intervals.
System Details
$ ssh -V
OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips 1 Mar 2016
$ echo $SHELL $TERM $DISPLAY
/bin/zsh xterm-256color :0
$ uname -r; cat /proc/version
4.4.0-22-generic
Linux version 4.4.0-22-generic (buildd@lgw01-41) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016
Observations
An odd observation is that the delay is there only when the allocation of a pseudo terminal occurs (i.e. the default, or the -t option). When the pseudo terminal is forcedly disabled, I can type commands and get their output without delay.
- My ~/.ssh/config file only contains "plain entries of the following format:
Host somename
HostName 192.168.1.0
User someuser
IdentityFile ~/.ssh/id_rsa
- There is no delay when I explicitly disable the pseudo terminal (i.e. using the -T option)
- Altering the settings mentioned in similar questions does not resolve the issue
- Ex: Top causes of slow ssh logins
- Most solutions I've found address the problem of slow login response. My issue occurs after I have logged in
- IntelliJ Idea Ultimate Edition can create SSH connections which do not have this issue. The IDE is running the Java SSH client JSch
- This has convinced me that it is a local issue and presumably not related to my network interface
- My coworkers on the same network do not have this issue, though they're using Macbooks
What I've Tried
- Enabling compressing, disabling X11 forwarding, various methods for speeding up the login process
- SSHing to localhost (no lag/delay here)
- SSHing to a variety of machines (i.e. not just one) -- delay is present
- Downloading the latest stable version of OpenSSH and building it for my system
- The freshly compiled binary has the same issue, leading me to think it's caused by some wonky configs
- Ensuring my /etc/ssh/ssh_config looks "normal"
- I've included it at the end of the post just in case
- Purging SSH and re-installing using apt-get
- Dropping into a virtual TTY (I think that's what they're called...) using Ctrl + Alt + F{1-6} and SSHing from there
- Rebooting
- Switching to a different network
- The lag/delay only seems to be an issue on the network at work. At home, there's no noticeable delay/lag
- Some other stuff that I'm forgetting
tl;dr
My SSH binary/config/something seems to have a delay when typing and I've tried a lot of things with no success. Maybe it's some character buffer thing? Who knows! Hopefully you do.
ssh -vvv
OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips 1 Mar 2016 debug1: Reading configuration data /home/redacted/.ssh/config debug1: /home/redacted/.ssh/config line 6: Applying options for ops1 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug2: resolving "xx.xx.xx.xx" port 22 debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to xx.xx.xx.xx [xx.xx.xx.xx] port 22. debug1: Connection established. debug1: identity file /home/redacted/.ssh/id_rsa type 1 debug1: key_load_public: No such file or directory debug1: identity file /home/redacted/.ssh/id_rsa-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to xx.xx.xx.xx:22 as 'redacted2' debug3: hostkeys_foreach: reading file "/home/redacted/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/redacted/.ssh/known_hosts:2 debug3: load_hostkeys: loaded 1 keys from xx.xx.xx.xx debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa,[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519 debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,[email protected],zlib debug2: compression stoc: none,[email protected],zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: host key algorithms: ssh-rsa,ssh-dss debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected] debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected] debug2: MACs ctos: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96 debug2: MACs stoc: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96 debug2: compression ctos: none,[email protected] debug2: compression stoc: none,[email protected] debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: aes128-ctr MAC: [email protected] compression: none debug1: kex: client->server cipher: aes128-ctr MAC: [email protected] compression: none debug3: send packet: type 34 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048 closed debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close logout debug3: channel 0: will not send data after close debug2: channel 0: obuf empty debug2: channel 0: close_write debug2: channel 0: output drain -> closed debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug3: send packet: type 97 debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1) debug3: send packet: type 1 debug1: fd 1 clearing O_NONBLOCK debug3: fd 2 is not O_NONBLOCK Connection to xx.xx.xx.xx closed. Transferred: sent 3376, received 3072 bytes, in 0.8 seconds Bytes per second: sent 3981.4, received 3622.9 debug1: Exit status 0
ssh -vvvT
OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips 1 Mar 2016 debug1: Reading configuration data /home/redacted/.ssh/config debug1: /home/redacted/.ssh/config line 6: Applying options for ops1 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug2: resolving "xx.xx.x.xx" port 22 debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to xx.xx.x.xx [xx.xx.x.xx] port 22. debug1: Connection established. debug1: identity file /home/redacted/.ssh/id_rsa type 1 debug1: key_load_public: No such file or directory debug1: identity file /home/redacted/.ssh/id_rsa-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to xx.xx.x.xx:22 as 'redacted' debug3: hostkeys_foreach: reading file "/home/redacted/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/redacted/.ssh/known_hosts:2 debug3: load_hostkeys: loaded 1 keys from xx.xx.x.xx debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa,[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519 debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,[email protected],zlib debug2: compression stoc: none,[email protected],zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: host key algorithms: ssh-rsa,ssh-dss debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected] debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-[email protected] debug2: MACs ctos: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96 debug2: MACs stoc: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96 debug2: compression ctos: none,[email protected] debug2: compression stoc: none,[email protected] debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: aes128-ctr MAC: [email protected] compression: none debug1: kex: client->server cipher: aes128-ctr MAC: [email protected] compression: none debug3: send packet: type 34 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048 drain debug2: channel 0: ibuf empty debug2: channel 0: send eof debug3: send packet: type 96 debug2: channel 0: input drain -> closed debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug2: channel 0: obuf empty debug2: channel 0: close_write debug2: channel 0: output drain -> closed debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug3: send packet: type 97 debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1) debug3: send packet: type 1 debug1: fd 1 clearing O_NONBLOCK debug3: fd 2 is not O_NONBLOCK Transferred: sent 3040, received 2800 bytes, in 1.9 seconds Bytes per second: sent 1598.6, received 1472.4 debug1: Exit status 0
/etc/ssh/ssh_config
# This is the ssh client system-wide configuration file. See # ssh_config(5) for more information. This file provides defaults for # users, and the values can be changed in per-user configuration files # or on the command line. # Configuration data is parsed as follows: # 1. command line options # 2. user-specific file # 3. system-wide file # Any configuration value is only changed the first time it is set. # Thus, host-specific definitions should be at the beginning of the # configuration file, and defaults at the end. # Site-wide defaults for some commonly used options. For a comprehensive # list of available options, their meanings and defaults, please see the # ssh_config(5) man page. Host * # ForwardAgent no # ForwardX11 no # ForwardX11Trusted yes # RhostsRSAAuthentication no # RSAAuthentication yes # PasswordAuthentication yes # HostbasedAuthentication no # GSSAPIAuthentication no # GSSAPIDelegateCredentials no # GSSAPIKeyExchange yes # GSSAPITrustDNS no # BatchMode no # CheckHostIP yes # AddressFamily any # ConnectTimeout 0 # StrictHostKeyChecking ask # IdentityFile ~/.ssh/identity # IdentityFile ~/.ssh/id_rsa # IdentityFile ~/.ssh/id_dsa # IdentityFile ~/.ssh/id_ecdsa # IdentityFile ~/.ssh/id_ed25519 # Port 22 # Protocol 2 # Cipher 3des # Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc # MACs hmac-md5,hmac-sha1,[email protected],hmac-ripemd160 # EscapeChar ~ # Tunnel no # TunnelDevice any:any # PermitLocalCommand no # VisualHostKey no # ProxyCommand ssh -q -W %h:%p gateway.example.com # RekeyLimit 1G 1h # SendEnv LANG LC_* HashKnownHosts yes # GSSAPIAuthentication yes # GSSAPIDelegateCredentials no
sshing into your localhost. What were the results of that? Same lag? No lag?