I'm trying to troubleshoot my home internet connection, which often hangs on large downloads. I believe I have been able to duplicate the problem between my home computer and an Amazon EC2 instance with packet captures at both ends. Here's the same behavior I've observed on two separate hangs:
- I start a TCP download of a 20MB file to my home computer from the EC2 server (home computer is TCP client, EC2 server is TCP server).
- The initial TCP SYN has an MSS option of 1418 bytes. No other segments from the client have an MSS option (the SYN/ACK from the server has MSS option 1460).
- When the bulk of the transfer gets going, the packets from the server are 2878 bytes. These packets are received on the client as two 1472 byte packets (I'll call them the front and back half), each containing a TCP segment with half the data.
- At some point a front half packet from the server is lost. The client starts repeating ACKs with a SACK option.
- Upon receiving the first ACK with a SACK option, the server starts sending 1472 byte packets (instead of 2878 byte packets).
- After a few duplicate ACKs, the server does Fast Retransmission, resending a 1472 byte packet - i.e. not the 2878 byte packet it originally sent. This resent packet does not show up at the client.
- After timeout, the server attempts Retransmission several more times, again sending the 1472 byte packet. None of these resent packets show up on the client.
I have several questions. First of all, why is the server sending 2878 byte packets to begin with? Isn't it supposed to honor the MSS option in the SYN packet?
Second, where are those 2878 byte segments being split into two 1472 byte segments?
Third, where and why are all the retransmitted packets being dropped?
Fourth, is there any way I can prevent this from the client side of things?
Interestingly, it appears that the connection can recover if a back half packet is lost. My (possibly bogus) theory is that something along the route is eating the retransmitted front half packets because they don't match their original size. Also, the same transfer tunneled through UDP (with OpenVPN) does not hang (I know that means I could proxy all my traffic this way but that's not the answer I'm looking for).
Followup: @DavidSchwartz suggests in the comments that the segment size anomalies could be explained by TSO, and it looks like that is correct. Disabling TSO on the server caps the segment size at 1472 bytes. Unfortunately, the transfer still hangs with retransmitted packets not being received.
I have also tried disabling TCP options SACK, timestamps, and window scaling, as well as reducing the window size to 8192 bytes. The transfer still hangs. The client receives packets sent before and after retransmitted packets but not the retransmitted packets themselves.