Ubuntu Server 14.04 - How to fix openssh “broken pipe” remotely?

Question

I just set up an Ubuntu Server and configured it via SSH using openssh on the server. I had to restart and cannot connect via SSH due to Write failed: Broken pipe error.

As far as my research goes (https://wiki.archlinux.org/index.php/SFTP_chroot - Troubleshooting, first entry), that's an issue with ownership problem with the ChrootDirectory. I likely messed something up while setting up the apache webserver.

Anyway, now I cannot login via SSH with my username. Unfortunatelly, I only have one user so I can't use a different one. Is there any way for me to fix ownership or the openssh settings without physical access to the server? Thanks!

Trying to connect with debug options shows:

ssh -v [email protected]
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 123.123.123.123 [123.123.123.123] port 22.
debug1: Connection established.
debug1: identity file /home/username/.ssh/id_rsa type -1
debug1: identity file /home/username/.ssh/id_rsa-cert type -1
debug1: identity file /home/username/.ssh/id_dsa type -1
debug1: identity file /home/username/.ssh/id_dsa-cert type -1
debug1: identity file /home/username/.ssh/id_ecdsa type -1
debug1: identity file /home/username/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/username/.ssh/id_ed25519 type -1
debug1: identity file /home/username/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.4
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.4 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr [email protected] none
debug1: kex: client->server aes128-ctr [email protected] none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA ...
debug1: Host '123.123.123.123' is known and matches the ECDSA host key.
debug1: Found key in /home/username/.ssh/known_hosts:7
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/username/.ssh/id_rsa
debug1: Trying private key: /home/username/.ssh/id_dsa
debug1: Trying private key: /home/username/.ssh/id_ecdsa
debug1: Trying private key: /home/username/.ssh/id_ed25519
debug1: Next authentication method: password
[email protected]'s password: 
debug1: Authentication succeeded (password).
Authenticated to 123.123.123.123 ([123.123.123.123]:22).
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
Write failed: Broken pipe

Answer 1

In my case the cause turned out to be an interaction between a VPN and a dodgy DSL modem with an external SSH port forward.

Scenario

Setup:

• 192.168.0.0/24 is the LAN network
• 192.168.1.0/24 is the VPN client network
• server1 (192.168.0.3) is the VPN server
• The DSL modem (192.168.0.10) is the default router for all hosts
• The DSL modem has a static route that sends VPN traffic (i.e. destined for 192.168.1.0/24) to server1
• In the DSL modem config, TCP port 22 is forwarded to server1
• In the DSL modem config, the VPN port is forwarded to server1
• server2 (192.168.0.223) is the server I'm trying to SSH to

The symptom was that during connection establishment, the client side always said "Write failed: Broken pipe" after the password or key verfication happened. This occurred right after the debug1: Sending command: ... step. I enabled detailed logging on the server, and it said the the client sent a TCP reset.

Analysis

I tested with Wireshark and found this this sequence occurred:

 23.002349   192.168.1.6 -> 192.168.0.223  TCP 56677 > ssh [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=643508643 TSER=0 WS=8
 23.080714  192.168.0.223 -> 192.168.1.6   TCP ssh > 56677 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1369 TSV=1628354 TSER=643508643 WS=7
 23.080743   192.168.1.6 -> 192.168.0.223  TCP 56677 > ssh [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=643508663 TSER=1628354
 23.399242  192.168.0.223 -> 192.168.1.6   SSH Server Protocol: SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u1\r
 23.399267   192.168.1.6 -> 192.168.0.223  TCP 56677 > ssh [ACK] Seq=1 Ack=40 Win=5888 Len=0 TSV=643508742 TSER=1628434
 23.399606   192.168.1.6 -> 192.168.0.223  SSH Client Protocol: SSH-2.0-OpenSSH_5.5p1 Debian-6+squeeze8\r
 23.479613  192.168.0.223 -> 192.168.1.6   TCP ssh > 56677 [ACK] Seq=40 Ack=42 Win=29056 Len=0 TSV=1628454 TSER=643508742
 23.479638   192.168.1.6 -> 192.168.0.223  SSHv2 Client: Key Exchange Init
 23.488855  192.168.0.223 -> 192.168.1.6   SSHv2 Server: Key Exchange Init
 23.528382   192.168.1.6 -> 192.168.0.223  TCP 56677 > ssh [ACK] Seq=890 Ack=960 Win=8960 Len=0 TSV=643508775 TSER=1628455
 23.607196  192.168.0.223 -> 192.168.1.6   TCP ssh > 56677 [ACK] Seq=960 Ack=890 Win=31872 Len=0 TSV=1628486 TSER=643508762
 23.607218   192.168.1.6 -> 192.168.0.223  SSHv2 Client: Diffie-Hellman GEX Request
 23.714323  192.168.0.223 -> 192.168.1.6   SSHv2 Server: Diffie-Hellman Key Exchange Reply
 23.714367   192.168.1.6 -> 192.168.0.223  TCP 56677 > ssh [ACK] Seq=914 Ack=1240 Win=10752 Len=0 TSV=643508821 TSER=1628512
 23.732326   192.168.1.6 -> 192.168.0.223  SSHv2 Client: Diffie-Hellman GEX Init
 23.837671  192.168.0.223 -> 192.168.1.6   SSHv2 Server: Diffie-Hellman GEX Reply
 23.876369   192.168.1.6 -> 192.168.0.223  TCP 56677 > ssh [ACK] Seq=1186 Ack=2088 Win=13568 L
 23.934571   192.168.1.6 -> 192.168.0.223  SSHv2 Client: New Keys
 24.052445  192.168.0.223 -> 192.168.1.6   TCP ssh > 56677 [ACK] Seq=2088 Ack=1202 Win=33664 Len=0 TSV=1628598 TSER=643508876
 24.052465   192.168.1.6 -> 192.168.0.223  SSHv2 Encrypted request packet len=52
 24.130296  192.168.0.223 -> 192.168.1.6   TCP ssh > 56677 [ACK] Seq=2088 Ack=1254 Win=33664 Len=0 TSV=1628617 TSER=643508906
 24.131032  192.168.0.223 -> 192.168.1.6   SSHv2 Encrypted response packet len=52
[...]
 29.775083   192.168.1.6 -> 192.168.0.223  SSHv2 Encrypted request packet len=136
 29.897679  192.168.0.223 -> 192.168.1.6   TCP ssh > 56677 [ACK] Seq=2728 Ack=2586 Win=39936 Len=0 TSV=1630059 TSER=643510336
 30.986259  192.168.0.223 -> 192.168.1.6   SSHv2 Encrypted response packet len=52
 30.986704   192.168.1.6 -> 192.168.0.223  SSHv2 Encrypted request packet len=136
 31.066976  192.168.0.223 -> 192.168.1.6   TCP ssh > 56677 [ACK] Seq=2780 Ack=2722 Win=41600 Len=0 TSV=1630351 TSER=643510639
 31.068851   192.168.0.10 -> 192.168.1.6   SSH Encrypted response packet len=88
 31.068874   192.168.1.6 -> 192.168.0.10   TCP 56677 > ssh [RST] Seq=1 Win=0 Len=0
 61.015948   192.168.1.6 -> 192.168.0.223  SSHv2 Encrypted request packet len=68
 61.094573  192.168.0.223 -> 192.168.1.6   TCP ssh > 56677 [RST] Seq=2780 Win=0 Len=0

What's going on here is that everything (some detail skipped) is fine up until the 31.066976 second mark (where server2 is sending an ACK for a request). But then, the next packet (at 31.068851) comes from THE DSL MODEM, which of course makes my laptop send it a TCP RST (connection reset) telling it that it's using an old connection or something. (I know this was related, because the RST is sent from port 56677.)

BUT, the DSL modem appears to be forwarding this reset back to server1, which of course kills the actual SSH connection. So, after a 30 second timeout (at 61.015948), my laptop tries to send another request having got no response on the actual TCP connection to the request at 30.986704. Naturally, server1 then sends it a TCP RST because it had already given up on that connection. Hence the Write failed: Broken pipe error on my laptop.

Conclusion

The DSL modem is seeing half of the connection, i.e. reply packets from server2 and forwarding them to server1 as VPN traffic. But at least some of these packets are being interpreted by the DSL modem in some fashion, because it's sending random responses (using SSHv1 btw). Note that the DSL modem can't see the packet at 30.986704, because that comes out of the tunnel on server1 and is sent via the LAN to server2. So I'm not really sure what's going on.

Workaround

I added a static route on server2 for 192.168.1.0/24 with gateway 192.168.0.3 (server1), which avoids reply traffic for VPN clients going via the DSL modem. Problem solved.

PS -- For those new to TCP, remember that separate ACK messages are only sent if there is no reply data available to send within a set interval; if there is, the ACK flag is just set on that data message.