1

I'm running on Ubuntu 17.04, openssh-client==7.4p1-10, kernel 4.10.0-33-generic.

I'm having problems with executing ssh commands, like:

rsync -t -e ssh -p 22 script.sh [email protected]:/var/lib/script.sh
\_ ssh -p 22 -l root [email protected] rsync --server -te.LsfxC . /var/lib/script.sh

it takes rsync 6 minutes to sync that script which has 4kB. The problem isn't just with rsync also git push over ssh sometimes gets sucked.

Funny thing is that after interrupting the process and executing it again it works immediately:

^Crsync error: unexplained error (code 130) at rsync.c(638) [sender=3.1.2]
rsync: [sender] write error: Broken pipe (32)

It doesn't seem to be a DNS issue, here's /etc/resolv.conf:

nameserver 8.8.8.8
nameserver 8.8.4.4
options single-request-reopen
options attempts:2
options rotate
options timeout:2

I've already disabled GSSAPI:

/etc/ssh/ssh_config:

   GSSAPIAuthentication no
   GSSAPIDelegateCredentials no

without any effect, I've tried forcing IPv4 connection with -4 also without any success. Any idea what could be wrong?

Here's strace of that process:

strace: Process 7610 attached
select(8, [3 5], [], NULL, NULL)        = 1 (in [3])
clock_gettime(CLOCK_BOOTTIME, {42870, 893598449}) = 0
read(3, "\372oyu\331J\20\327\264\325\357\274\vn\233\nG\207\207c\251\230\341NzUk\261\351v\23\353"..., 8192) = 44
clock_gettime(CLOCK_BOOTTIME, {42870, 894108136}) = 0
clock_gettime(CLOCK_BOOTTIME, {42870, 894258960}) = 0
select(8, [3 5], [6], NULL, NULL)       = 1 (out [6])
clock_gettime(CLOCK_BOOTTIME, {42870, 894325845}) = 0
write(6, "\3\0\0\7\0\0\0", 7)           = 7
clock_gettime(CLOCK_BOOTTIME, {42870, 894439661}) = 0
clock_gettime(CLOCK_BOOTTIME, {42870, 894473071}) = 0
select(8, [3 5], [], NULL, NULL)        = 1 (in [5])
clock_gettime(CLOCK_BOOTTIME, {42870, 894558087}) = 0
read(5, "\2\0\0\7\0\0\1\0\0\7\0", 16384) = 11
clock_gettime(CLOCK_BOOTTIME, {42870, 894661575}) = 0
clock_gettime(CLOCK_BOOTTIME, {42870, 894699595}) = 0
select(8, [3 5], [3], NULL, NULL)       = 1 (out [3])
clock_gettime(CLOCK_BOOTTIME, {42870, 894780961}) = 0
write(3, "\f\16\6UF|B\1\315\nYP\355\f|\177|\234v\371\322\236*)\32`\3214\225$u\337"..., 52) = 52
clock_gettime(CLOCK_BOOTTIME, {42870, 894852781}) = 0
clock_gettime(CLOCK_BOOTTIME, {42870, 894874370}) = 0
select(8, [3 5], [], NULL, NULL)        = 1 (in [3])
clock_gettime(CLOCK_BOOTTIME, {42870, 923152465}) = 0
read(3, "\310\3258\332\212)\re\262\322^\f\275\324X{\361\23f\211mk'\213\224\v\0\204\322\n\25\221"..., 8192) = 44
clock_gettime(CLOCK_BOOTTIME, {42870, 923618233}) = 0
clock_gettime(CLOCK_BOOTTIME, {42870, 923845130}) = 0
select(8, [3 5], [6], NULL, NULL)       = 1 (out [6])
clock_gettime(CLOCK_BOOTTIME, {42870, 923946992}) = 0
write(6, "\1\0\0\7\0", 5)               = 5
clock_gettime(CLOCK_BOOTTIME, {42870, 924002335}) = 0
clock_gettime(CLOCK_BOOTTIME, {42870, 924027449}) = 0
select(8, [3 5], [], NULL, NULL)        = 1 (in [3])
clock_gettime(CLOCK_BOOTTIME, {42870, 943180384}) = 0
read(3, "\326U\32\20\246\374\201K\246\177!z\265\302^\252\371\255\215\355\265\356\313\322W\2341`%\215\20P"..., 8192) = 176
close(6)                                = 0
close(5)                                = 0
clock_gettime(CLOCK_BOOTTIME, {42870, 943307191}) = 0
clock_gettime(CLOCK_BOOTTIME, {42870, 943334146}) = 0
close(7)                                = 0
select(8, [3], [3], NULL, NULL)         = 1 (out [3])
clock_gettime(CLOCK_BOOTTIME, {42870, 943414987}) = 0
write(3, "0\236\27\233p\303\324\302\222mD\242Y_\34S\365\366p\214z\320\367.sN\252\337\322S\202("..., 36) = 36
rt_sigaction(SIGWINCH, NULL, {0x5639600b7460, [], SA_RESTORER, 0x7f7046de37f0}, 8) = 0
rt_sigaction(SIGWINCH, {SIG_DFL, [], SA_RESTORER, 0x7f7046de37f0}, NULL, 8) = 0
write(3, "F\226\207\7\243\207\33\316\37\1U$\326Y\314\253\310p\210\354\240\247\322n\32\272A\312\312:\252\324"..., 60) = 60
ioctl(0, TCGETS, 0x7ffc20de6720)        = -1 ENOTTY (Inappropriate ioctl for device)
fcntl(0, F_GETFL)                       = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(0, F_SETFL, O_RDWR)               = 0
ioctl(1, TCGETS, 0x7ffc20de6720)        = -1 ENOTTY (Inappropriate ioctl for device)
fcntl(1, F_GETFL)                       = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(1, F_SETFL, O_RDWR)               = 0
ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0
shutdown(3, SHUT_RDWR)                  = 0
close(3)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Other thing I've noticed is relatively high number of retransmits (just few minutes after booting the system) - other devices in the same network works fine. A malfunctioning network card?

$ netstat -s | egrep -i 'loss|retran'
    421 segments retransmitted
    TCPLostRetransmit: 6
    1 timeouts in loss state
    47 fast retransmits
    137 retransmits in slow start
    TCPLossProbes: 7
    TCPRetransFail: 3
    TCPSynRetrans: 12

EDIT:

I've already tried without any success:

  • replacing network cable (goes directly to router)
  • replacing NIC card (on board Broadcom by Realtek gigabit card)
6
  • in most cases when i have this problem - firewall / iptables configuration is the reason. you can check this by disabling for a moment firewall on both ends.
    – rsm
    Jul 25, 2017 at 13:13
  • @rsm I don't think the connection could be blocked by firewall when running the same command twice has same output. Except first run takes 6 minutes and the other one 3 seconds.
    – Tombart
    Jul 25, 2017 at 13:17
  • there is something more happening in the background when ssh is establishing connection and it takes sometimes (especially at the beginning) couple of minutes. then for some time it works ok. it looks mysterious, but as i said, when i have this problem first place i check is firewall config (not only ssh, but also dns etc rules). disabling firewalls on both ends for a moment is the simplest test.
    – rsm
    Jul 25, 2017 at 14:43
  • @rsm I've tried disabling firewall it doesn't change anything
    – Tombart
    Aug 8, 2017 at 21:57
  • How about at the SSH server level and not the client level? What do you control and maintain at this level? Can you see SSH remote server logs? Can you disable FW rules on the server or other applicable ACL rules, filters, or proxies along the way across the network? I'm a Linux noob so perhaps I'm overlooking something above you mention that goes over the remote server level items. Aug 8, 2017 at 22:40

2 Answers 2

1

You can get more debug information by trying a simple ssh -vvv to the server and see the messages coming from the client process.

Also try to telnet to the ssh port (22 by default) and see how fast it will respond.

As the others have suggested it might be a firewall problem (seems like limits for incoming connections) however, since you have disabled it and it didn't help much it might not be the case this time.

Another option is user / group information that holds the connection for quite a while, for example when connecting with a machine that uses remote LDAP server and it's busy or is having trouble reaching the LDAP (which is needed to resolve your uid/gid) it will also delay the connection. (if possible try logging into root account with ssh key as it should not use external servers)

Yet another thing to check is the DNS server on the remote end, ssh server might try to resolve your IP address into DNS host and if it's DNS server is unreliable it might also take some time to do it.

As for the connections following the first one and being much faster it might also indicate the problem being in some kind of caching mechanism (DNS, LDAP, netfilter RELATED,ESTABLISHED state), or simply that your ssh client uses controlsockets (and it keeps them open after initial connection)

1
  • I've already tried ssh -vvv, same for rsync without any result. It gets stuck at random file. It can't be DNS issue (I've added resolv.conf) there's no way DNS lookup would take 6min with 2s timeouts and 2 retries with only 1 query to do. I'm not using LDAP on any side. ssh authentication is done using RSA key.
    – Tombart
    Aug 9, 2017 at 6:30
1

After few unsuccessful attempts, I've tuned network related parameters /etc/sysctl.conf with following values:

net.core.netdev_max_backlog = 5000
# allow testing with buffers up to 64MB
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
# increase Linux autotuning TCP buffer limit to 32MB
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
# recommended default congestion control is htcp
net.ipv4.tcp_congestion_control=htcp
# recommended for hosts with jumbo frames enabled
net.ipv4.tcp_mtu_probing=1
net.core.default_qdisc = fq

Increasing just TCP buffers didn't help. Now the network is behaving as expected.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .