Why do I only get 300-400 Mbit/s on my 1000 Mbit/s network?

Question

My setup is like this:

Laptop <--> Router <--> Desktop

They all support gigabit Ethernet. I'm using the 'iperf' TCP/IP benchmark on the laptop and desktop, and I've never gotten more than 400 Mbit/s raw throughput. The laptop is an Intel Core 2 Duo 2 GHz running Windows XP, the desktop is an Intel Core 2 quad 2.4 GHz running Windows 7. What things can I look at to maximize the network speed between them? Software settings? A way to know if my cable is not up to specifications? How to know if the gigabit Ethernet controller in one of the devices just isn't capable of hitting the max speed?

Answer 1

1 Gbit/s is the theorectical maximum of this connection.

Many things can affect this.

Expect a hit in performance from your on-board NICs, as they will most likely be (based on device descriptions "Laptop" and "Desktop") host-based adapters that rely on the CPU to process the network traffic. Therefore speed will be affected by the CPU doing other things (like getting the data from the HDDs to the networking sub-system for example).

Also, depending on what the "router" is, it may be trying to process the data as it passes from LAN port to LAN port, so its processing speeds may also be affecting the data transfer speeds, regardless of how fast its 'ports' are.

If I were you my next test would be to get myself a 'just long enough' CAT6 crossover patch cable and wire the notebook directly to the PC (you may be able to get away with a straight-through cable, as many NIC's support auto-crossover these days).

That way you (at least) can pretty easily eliminate or identify the router (and/or existing wiring) as a bottleneck.

Answer 2

iperf uses a pretty small window size by default.

Increase the window or run it in UDP mode and it will easily saturate a 1 Gbit/s connection.

Answer 3

Jeff Atwood has some good analysis of why real-world throughput of a Gigabit network is closer to, in his calculations, 30MB/s. (big 'B', not little 'b')

...you definitely shouldn't expect the perfect scaling we achieved moving from 10baseT to 100baseT. Without any major tweaking, you'll get only a fraction of the tenfold bandwidth improvement you might expect

Answer 4

If everything is gigabit, you could try to enable jumbo frames. One of my friends did this with his home computer and noticed a huge increase in throughput. However, there are some drawbacks for low-latency applications like VOIP, as mentioned in the article.

Answer 5

Did you say you're using a router? is that an off the shelf consumer grade router (not switch)?

I'd say thats most likely your bottleneck. To verify this, Plug the laptop directly into the desktop. Set your Laptop and desktop to use a static IP address like 192.168.1.10 (laptop), 192.168.1.11 (desktop).

This should allow the laptop to talk directly to the desktop. Run iperf between them and report the result back here.

That should work as most modern network cards can auto-sense transmit/receive wires unlike the old days when you used to have to build a crossover cable.

If you see a marked increase then your router is the culprit. Some of them have switches on the back, others can actually route between ports. Many of these routers are terrible performers. If you do find this is the problem then if I were you I'd either upgrade the router (which is probably fine for internet use) or go and buy yourself a GigE switch and cascade that onto the router switch. Plug your laptop and desktop into the switch. That way, they will bypass the router to talk to each other.

Make sure you change your laptop & desktop back to DHCP when you've completed the experiment.

To give you an idea, you should get I would think 650Mbit/s and above. iperf doesn't use the harddrive unless you tell it to. So it's memory to memory or really testing the network.

On of the other bottlenecks is actually how quick your CPU can transfer data from the memory to the network card. Networks that are on PCI express seem to operate better. I think laptops are generally fairly cheaply built these days so if you don't see much above that then this could be the other bottleneck.

Also, consumer grade switches can often only cope switching 1GB between any two ports whereas high end managed switches can switch at wire speed on all ports simultaneously. That's part of the reason they cost so much more.

Answer 6

We've performed numerous test and it boiled down to the Operating System.

If we go from Ubuntu 11.04 to Ubuntu 11.04 (same with CentOS for Linux machines), the speed throughput is ~938-941 MBits/s constant. If we go from Windows 7 x64 SP1 Ultimate to Ubuntu 11.04, it varies from ~440-475 MBits/s. If we move from Windows 2008 R2SP1 x64 to Ubuntu 11.04, the speed measures ~435-635 MBits/s.

As you can see, Linux based machines fully utilize the Gigabit links compared to Windows. We are now investigating settings in Windows that would result in the same throughput as Linux on same hardware. Hard drive limitations are not a factor since this is in memory. For the record, the machines we used had 7200 RPM drives with SATA 3.0. Both the Ubuntu 11.04 (or Linux base) and Windows exceeded the theoretical speed of those SATA links.

Answer 7

Transfer speeds are only as good as the weakest link in the chain. It might not be the network speed but the write/read speed of your Hard Drives.

You can download RamDisk software and create a RamDisk on both computers. Place a decent size file inside of it and do a transfer over the network between the RamDisks. This should show you the actual speed of your network that won't be bottlenecked by your Hard Drive speeds.

Answer 8

Not a direct answer to your question but recently we got the following result with iperf between two Linux boxes (8 core, 2.66 GHz each):

------------------------------------------------------------
Client connecting to 192.168.74.15, TCP port 5001
TCP window size:  192 KByte (default)
------------------------------------------------------------
[  3] local 192.168.74.25 port 54016 connected with 192.168.74.15 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec

with MTU=1500 (i.e. not using jumbo frames). That should give you an idea of what performance is in principle possible.

The network cards (lspci output) are:

82571EB Gigabit Ethernet Controller

on both sides.

You might want to check the CPU load on the server and especially on the client side and see whether this is close to 100%. Even though I have to say that on our system the busiest core is 80% (83%) idle on the receiver (sender) side while the other 7 seem to be 100% idle (checked with mpstat).

This is running iperf without the -u option, i.e. this uses TCP.

Answer 9

davr, Unlike fibre channel and similar technologies which use sophisticated bandwidth allocation schemes, ethernet is a 'listen and squawk' protocol. Ethernet first listens if anyone is currently transmiting, if so wait if not then send. The problem is that if two or more parties wishing to transmit are listening at the same time, they'll send at the same time as well! This will cause collisions and retransmits. Fancy switches eliminate a lot of this, but not all.

With ethernet you should expect see about 30-40 percent bandwidth (Looks like your getting this . . . I could be wrong on the exact #, much depends on switch quality and etc) before lots of collisions start to occur, with fabric technology is something like 80%. Ethernet is cheap, fibre channel is not . . . I'm not sure what 4G ethernet does to get around this . . . it might a have a more 'fabric' like network topology similar to fibre channel and friends. This is an oversimplification, but is essentially correct.