Throughput decreases after MTU 5000

Question

I am trying to test throughput between two computers directly connected through 1 GbE and testing with iperf. I am getting a throughput around 980Mbps when MTU is between 5000 and 5050, however it drastically falls to around 680Mbps anything above MTU=5050. I have checked varying window sizes, but with same result. Increasing the MTU should decrease the overheads and thereby should increase the bandwidth or at least should not fall. I can't figure out this strange behavior. By the way testing TCP throughput. Any help ! and thanks guys. This is my post ever on any forum :) usually I find answers....

Additional info! Two centos systmes, One of them is Xen 4.2 host (but it shouldn't be the problem) Checked with varying buffer sizes in /pro/sys/net/ipv4, but no same result. Latency is 0.2 ms.

Answer 1

This is almost certainly an implementation specific performance bottleneck somewhere. You're correct in that all other things equal having larger MTUs will result in lower overhead and thus higher performance. But other things are rarely equal... here, some part of the forwarding/packet moving code probably has a base buffer size of 5KB or something like that, so when you cross that barrier, all of a sudden you're asking the system to do twice as much work.

One way to support this theory would be to increase the MTU even further. If you see a big drop at 5KB, but then your thruput improves once beyond that, you've almost certainly hit some buffer size threshold somewhere in the code path.