0

I'm trying to troubleshoot my home internet connection, which often hangs on large downloads. I believe I have been able to duplicate the problem between my home computer and an Amazon EC2 instance with packet captures at both ends. Here's the same behavior I've observed on two separate hangs:

  • I start a TCP download of a 20MB file to my home computer from the EC2 server (home computer is TCP client, EC2 server is TCP server).
  • The initial TCP SYN has an MSS option of 1418 bytes. No other segments from the client have an MSS option (the SYN/ACK from the server has MSS option 1460).
  • When the bulk of the transfer gets going, the packets from the server are 2878 bytes. These packets are received on the client as two 1472 byte packets (I'll call them the front and back half), each containing a TCP segment with half the data.
  • At some point a front half packet from the server is lost. The client starts repeating ACKs with a SACK option.
  • Upon receiving the first ACK with a SACK option, the server starts sending 1472 byte packets (instead of 2878 byte packets).
  • After a few duplicate ACKs, the server does Fast Retransmission, resending a 1472 byte packet - i.e. not the 2878 byte packet it originally sent. This resent packet does not show up at the client.
  • After timeout, the server attempts Retransmission several more times, again sending the 1472 byte packet. None of these resent packets show up on the client.

I have several questions. First of all, why is the server sending 2878 byte packets to begin with? Isn't it supposed to honor the MSS option in the SYN packet?

Second, where are those 2878 byte segments being split into two 1472 byte segments?

Third, where and why are all the retransmitted packets being dropped?

Fourth, is there any way I can prevent this from the client side of things?

Interestingly, it appears that the connection can recover if a back half packet is lost. My (possibly bogus) theory is that something along the route is eating the retransmitted front half packets because they don't match their original size. Also, the same transfer tunneled through UDP (with OpenVPN) does not hang (I know that means I could proxy all my traffic this way but that's not the answer I'm looking for).

Here are packet captures for home and server.

Followup: @DavidSchwartz suggests in the comments that the segment size anomalies could be explained by TSO, and it looks like that is correct. Disabling TSO on the server caps the segment size at 1472 bytes. Unfortunately, the transfer still hangs with retransmitted packets not being received.

I have also tried disabling TCP options SACK, timestamps, and window scaling, as well as reducing the window size to 8192 bytes. The transfer still hangs. The client receives packets sent before and after retransmitted packets but not the retransmitted packets themselves.

2
  • 2
    It's possible that your capture of the packets from the server is showing the packets before the NIC while the packets after the NIC are different due to TSO. Nov 20 '14 at 2:48
  • It looks like you're right - if I disable TSO on the server then the packets are all 1472 bytes. The transfer still hangs but at least I can write off the segment size anomaly as a red herring.
    – rhashimoto
    Nov 20 '14 at 17:07
1

I know this is a very old post but so far the two answers didn't address the disabling of the options. By disabling Window Scaling and SACK you've gimped TCP significantly. That by itself will cause throughput issues. However, you mentioned that after a timeout the server re-transmits several more times. This is the problem. RTO has kicked off the exponential back off. What that means is the other end of the connection is completely unavailable for an extended period of time. This problem is a significant one and not a TCP issue because TCP is doing what its designed to do. But more likely a hardware issue with a networking device like a router.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.