Weird slow uplink problems on my home Debian router and new fast connection

Question

Yesterday I got a new and shiny VDSL2 connection home! It's specced at 100Mbit/10Mbit, and seems to deliver pretty close to the mark.

Now, I have a Debian squeeze linux box acting as a home NAS and router. It's running shorewall, with NAT and tc enabled. I also have an OSX workstation connected via a switch to said linux router:

OSX Workstation <-> Switch <-> Debian router <-> VDSL2 Modem <-> Internet <-> Server

I ran tests against my fast server on the internet:

On Linux router, TCP:

$ iperf -c server -p 3333
------------------------------------------------------------
Client connecting to server, TCP port 3333
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[  3] local xxx.yyy.bbb.ccc port 41982 connected with xxx.yyy.bbb.ccc port 3333
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.89 MBytes  2.42 Mbits/sec

On the above is where the problem lies. The uplink should be ~10Mbit, not 2.4Mbit. Below, you can see that UDP is working fine.

On Linux router, UDP:

$ iperf -u -c server -p 60008 -b 9M
------------------------------------------------------------
Client connecting to server, UDP port 60008
Sending 1470 byte datagrams
UDP buffer size: 1.00 MByte (default)
------------------------------------------------------------
[  3] local xxx.yyy.bbb.ccc port 56484 connected with xxx.yyy.bbb.ccc port 60008
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.7 MBytes  9.00 Mbits/sec
[  3] Sent 7658 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec  10.7 MBytes  9.00 Mbits/sec  0.251 ms    0/ 7657 (0%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order

And on the OSX Workstation (behind NAT) TCP:

$ iperf -c server -p 3333
------------------------------------------------------------
Client connecting to server, TCP port 3333
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[  5] local 192.168.9.141 port 54388 connected with xxx.yyy.bbb.ccc port 3333
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  13.2 MBytes  11.1 Mbits/sec

The OSX behind the linux router seems to be unaffected by the problems at the linux router. How can this happen? UDP works fine, too.

And on the OSX Workstation (behind NAT) UDP:

$ iperf -u -c server -p 60008 -b 9M
------------------------------------------------------------
Client connecting to server, UDP port 60008
Sending 1470 byte datagrams
UDP buffer size: 9.00 KByte (default)
------------------------------------------------------------
[  5] local 192.168.9.141 port 64588 connected with xxx.yyy.bbb.ccc port 60008
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  10.7 MBytes  9.00 Mbits/sec
[  5] Sent 7658 datagrams
[  5] Server Report:
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  5]  0.0-10.0 sec  10.7 MBytes  9.00 Mbits/sec  0.133 ms    0/ 7658 (0%)

As you can see, the linux box is stuck at 2.5Mbit/s outbound TCP. UDP works fine, and the workstation behind the router works fine.

To simplify the situation, I modified my Shorewall TC to a very basic level. I also tried turning TC off alltogether from shorewall without any effect. :

tcdevices:

#INTERFACE  IN-BANDWITH OUT-BANDWIDTH
eth0        -           12000kbit

tcclasses:

#INTERFACE      MARK    RATE            CEIL        PRIORITY    OPTIONS
eth0            1       full            full        1           default

tcrules:

#MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT   USER
1:F             0.0.0.0/0       0.0.0.0/0       icmp    echo-request
1:F             0.0.0.0/0       0.0.0.0/0       icmp    echo-reply

Do you have any idea where the problem might be? The only non-default thing I'm running on Debian is a 3.2.0 kernel from backports. The box is a powerful Xeon machine with lots of RAM and Intel network cards. All the tests were done in a short timeframe with practically no other network traffic. And repeated multiple times. Where could I start debugging?

Answer 1

A couple points:

1.) TCP may take a bit to ramp up to full speed - both in terms of finding the appropriate window size on longer flows as well as the setup time on shorter flows (i.e. 1.5 x RTT doing handshakes w/TCP vs immediate start with UDP). 10 megabytes isn't very much data. As you push toward the 1G mark you should see the average speed of TCP improve.

2.) Even when fully optimized there's a trade-off between reliability and performance. TCP can be inordinately sensitive to microbursts even in perfect lab conditions. Running over a DSL setup (no matter how fast) is going to be potentially be exposing the traffic flow to variable amounts of buffering, serialization delay, etc. Again, this tends to average out over longer flows. UDP (especially in performance benchmarks) will transmit as hard as it can, without regard for packet loss, buffer overflow, etc.

So the idea with TCP windowing is that if it's operating perfectly then we reach a situation where there's an optimal amount of traffic in flight with the smallest possible percentage of overhead (i.e. acknowledgements, TCP headers, etc). Indeed, the amount of traffic in transit should allow for a constant amount of bandwidth - even will acknowledgements are flowing. TCP's performance will asymptotically approach a wide open transmission of data.

UDP, in contrast, is a wide open transmission of data...

Now - the difference in performance between the various platforms is going to be a function of the particular implementation of TCP as well as how the system itself is prioritizing I/O and protocol processing. There may be a tweak or two that will bring the two values into better parity.. The size of the sample is still going to be an issue, though. The more tests and the larger the size, the more accurate the results.

Answer 2

I found the solution to the problem. All I had to do was turn off segmentation offloading on the network card:

ethtool -K eth0 gso off tso off

This fixed the problem for me. Apparently it's fairly common.