I got two desktop computers talking to each other directly. They both have Gigabit Ethernet capable network adapters. That's 1 Gbps or 1000 Mbps. I have them connected with a brand new 10 meter long Cat6 UTP straight cable and I get pretty close to that theoretical maximum. The Windows Task Manager (Network tab) shows 844 - 946 Mbps in one direction. But in the other direction, it only shows about 326 - 365 Mbps.
Local: 192.168.100.152 Remote: 192.168.100.151
The local computer runs Windows 8.1 Pro, and I have it connected remotely to the other computer which runs Windows Vista Ultimate.
I used Iperf to do some testing. I ran the test for 60 seconds each time. I ran the test 10 times for each direction of communication. I have then put together this table with test results to get an average.
192.168.100.152 -> 192.168.100.151 106 MB/s 192.168.100.152 -> 192.168.100.151 107 MB/s 192.168.100.152 -> 192.168.100.151 108 MB/s 192.168.100.152 -> 192.168.100.151 107 MB/s 192.168.100.152 -> 192.168.100.151 107 MB/s 192.168.100.152 -> 192.168.100.151 104 MB/s 192.168.100.152 -> 192.168.100.151 101 MB/s 192.168.100.152 -> 192.168.100.151 108 MB/s 192.168.100.152 -> 192.168.100.151 108 MB/s 192.168.100.152 -> 192.168.100.151 108 MB/s ---------------------------------------------------- Min: 101 MB/s Max: 108 MB/s Avg: 106.4 MB/s (851.2 Mbps) 192.168.100.152 <- 192.168.100.151 41.1 MB/s 192.168.100.152 <- 192.168.100.151 41.1 MB/s 192.168.100.152 <- 192.168.100.151 41.1 MB/s 192.168.100.152 <- 192.168.100.151 41.0 MB/s 192.168.100.152 <- 192.168.100.151 41.0 MB/s 192.168.100.152 <- 192.168.100.151 41.1 MB/s 192.168.100.152 <- 192.168.100.151 41.0 MB/s 192.168.100.152 <- 192.168.100.151 41.1 MB/s 192.168.100.152 <- 192.168.100.151 41.1 MB/s 192.168.100.152 <- 192.168.100.151 41.1 MB/s ----------------------------------------------------- Min: 41.0 MB/s Max: 41.1 MB/s Avg: 41.07 MB/s (328.56 Mbps)
My question is, why is it so much slower in the other direction?
Windows Task Manager
This is the network diagram as seen while performing the tests in Iperf.
Pay attention to the diagram in the following two screenshots!
Did you notice how it changed from "1 Gbps" to "500 Mbps" in the upper right corner as I switched from sending data to receiving data. Why did it do that? Is it somehow sensing the other network port as half of 1 Gbps when going one way, but yet full when going the other way around?
File transfer test
I did some more testing with a data file, to get some more realistic disk to disk readings. I created a 1 GB file for this purpose. I used only default Windows file sharing facilities. From the local computer, I connected to the C$ share on the remote computer and dragged and dropped the file back and forth (rope skipping) between them, changing the file name each time. I timed everything to the best of my abilities and this is what I got.
192.168.100.152 -> 192.168.0.151 1073741824 Byte 25 s 40,96 MB/s 192.168.100.152 -> 192.168.0.151 1073741824 Byte 20 s 51.2 MB/s 192.168.100.152 -> 192.168.0.151 1073741824 Byte 16 s 64 MB/s 192.168.100.152 -> 192.168.0.151 1073741824 Byte 16 s 64 MB/s 192.168.100.152 <- 192.168.0.151 1073741824 Byte 11 s 93.091 MB/s 192.168.100.152 <- 192.168.0.151 1073741824 Byte 34 s 30.118 MB/s 192.168.100.152 <- 192.168.0.151 1073741824 Byte 11 s 93.091 MB/s 192.168.100.152 <- 192.168.0.151 1073741824 Byte 11 s 93.091 MB/s
The throughput indicated by the Windows file copy diagram is telling a different story. Here I am downloading two files, one after the other to two different locations on the same disk. The first copy indicated 107 MB/s sustained up to 41%, and the second one indicated 98.9 MB/s sustained up to 87%.
So this is in line with the results I got with the Iperf tool. Now, here is what it looks like when I upload to the remote computer.
It sustains 103 MB/s up to 73%, and then it poops down to stinking 27.3 MB/s at 82% and then goes up a notch to reach 49.1 MB/s at 93%.
Here are two more funny looking "roller coaster" diagrams.
Update 1 - Link speed
I tried disabling the Wifi adapter on the remote computer. (The Wifi adapter was already disabled on the local computer.) I think this is what Timtech meant by that comment. I had the same thought - that having both wired and wireless adapters enabled at the same time limited the throughput on the wired adapter to the level of the Wifi adapter (adapting to the slowest adapter for compatibility). Because the Wifi adapter (DWA-160 Wireless N in this case) is usually detected as a "52 Mbps" - "104 Mbps" link by the Vista computer.
In the following screenshot, the remote computer is set up as a server and the local computer is set up as a client (192.168.100.152 <- 192.168.100.151).
But disconnecting the Wifi adapter on the remote computer did not help my low throughput on my wired connection.
Not only that! In Windows Task Manager on the remote computer, the link speed for the wired adapter (LAN 1) appears as "1 Gbps". If you refer to the screenshots above you can see that it is detected as a "500 Gbps" link on the local computer. So for the same wired connection, Windows Vista says it's a 1 Gbps link, while at the same time, Windows 8.1 Pro says it's a 500 Gbps link... which one is right then?
Here's what it looks like on the remote computer when I set it up as a client and the local computer as a server (192.168.100.152 -> 192.168.100.151).
As you can see here, about 95% of the 1 Gbps link is being utilized. That translates to 950 Mbps. It's exactly what I got in the test above. But going the other way around is a whole different story.
Update 2 - Duplexing and MDI-X
As suggested by some of you I had a look at the duplex settings. Both local and remote computer were set to auto negotiation mode, as you can see by the screenshots below.
I tried changing to "1.0 Gbps Full duplex" on both computers. I then did the same type of tests as I did before, using Iperf. With local computer as server and remote computer as client I get around 950 Mbps max. With local computer as client and remote computer as server I get around 360 Mbps.
Here, have a look at these screenshots.
What you see here is the diagram for when I upload and download between the two computers. The higher graph (95 - 98% utilization) is local to remote (upstream 192.168.100.152 -> 192.168.100.151). The lower graph (~ 33% utilization) is remote to local (downstream 192.168.100.152 <- 192.168.100.151).
To try to rule out any Auto MDI-X issues, I had one of those crossover adapters attached to one end of the cable (the local computer).
That would certainly make the cable a crossover cable. Hell, I even had it tested with a network tester! It is indeed crossed now (pins 1/3, 2/6)!
So now I have true crossover cable connection between the two computers, and I have manually set "1.0 Gbps Full duplex". Yet I still have the same problem. Any more ideas? Besides upgrading the Vista computer (or reinstalling the 8.1 computer)?
Update 3 - Software or Hardware limitation?
My best guess is that I have two operating systems that are not compatible with each other. They are both Windows systems, but not all Windows systems are made equal. I will have to try using Vista on both, or 8.1 Pro on both and see what kind of throughput I get.
That means buying an upgrade. Damn you Microsoft.
Both computers are custom built by the way. Here are some specs.
Local ----- Gigabyte GA-EP45-UD3R Intel Core 2 Quad Q9650 Intel P45 Corsair XMS2 DHX TwinX DDR2 PC6400/800MHz 4 GB Realtek 8111C chips (10/100/1000 Mbit) WD Caviar Black WD1002FAEX Windows 8.1 Pro 64-bit Remote ------ Gigabyte GA-X38-DQ6 Intel Core 2 Duo E4500 Intel X38 Corsair XMS2 DHX TwinX DDR2 PC6400/800MHz 4 GB Dual Realtek 8111B chip (10/100/1000 Mbit) WD Caviar Black WD1002FAEX Windows Vista Ultimate 64-bit
Tonny suggested that the Vista machine might be using a bad Realtek chip. So I dug up these specs. I see now that the Vista machine uses a B revision of 8111 while the local machine uses a C revision of the same chip. Does this mean anything? They are both clearly specified for 1000 Mbit (see above) by the manufacturer. Could it be that 8111B underperforms that much (360 Mbps)?
These particular drives reach 107 MB/s burst rate. That's exactly the number I've seen in the test on the local computer. But even the sustained sequential or random read/write of maybe 55 MB/s does NOT translate to 360 Mbps. That should give me somewhere around 440 Mbps, and not the 360 Mbps I'm getting. So I don't suspect that to be the bottleneck, especially since they both use the same drive model. And besides, file copy operation is one thing, but Iperf is not using disks at all, it only uses RAM memory for the tests.
Update 4 - TCP Checksum Offload
As suggested by Tonny, I tried switching off TCP checksum offloading (for IPv4 and IPv6).
I also switched "Speed & Duplex" back to auto for both computers. But this did not help. I still have the low throughput in one direction, and high in the other direction.
Update 5 - New driver version
I tried updating the driver version on both local and remote to the latest version downloaded from Gigabyte website and Realtek website.
Update path... On local (RTL8111C): 8.1.510.2013 (2013-05-10, Microsoft) 8.20.815.2013 (2013-08-15, Realtek) On remote (RTL8111B): 6.241.623.2010 (2010-06-23, Realtek) 6.250.908.2011 (2011-09-08, Realtek) 6.252.1109.2012 (2012-11-09, Realtek)
I still got the same lousy throughput in one direction.
Update 6 - CPU utilization
I checked CPU utilization. This should not be an issue. Here are my findings.
On local... Download: 4 - 10 % Upload: 4 - 10 % Idle: 0 - 4 % On remote... Download: 24 - 38 % Upload: 10 - 25 % Idle: 1 - 6 %
Local (download, upload, idle)...
Remote (download, upload, idle)...
The remote uses much more CPU power, but this is also the one with the slower Core 2 Duo. But it never exceeded the 38 % point during my tests. What's particularly interesting here is that it uses so much more CPU power when downloading (local -> remote) than when uploading (local <- remote).
So with a throughput of 950 Mbps it uses 38% and at 360 Mbps it uses 25 %. Also the core utilization is not balanced, it uses one core more than the other. I'm not sure what conclusion to draw from this. The local computer doesn't display core utilization, so I can't compare it. But CPU utilization is even on the local computer (10 % on download/upload).
Update 7 - New Intel Gigabit network adapter
I have now installed a brand new PCI-Express Gigabit network adapter from Intel as a replacement for the built-in Realtek RTL8111B on the remote computer, which is supposedly too slow on upload. The product number of the Intel adapter is EXPI9301CT. This adapter is supposed to be very good according to reviews I've read. I just want to rule this out as a possible bottleneck.
I've done some testing now with Iperf for Windows and here are the results.
Local (download, upload)...
Remote (download, upload)...
On average, this adapter is actually a little bit slower than the Realtek adapter. I think it has a smaller overhead than the Realtek and as a result of that a more stable continuous throughput. But I still get only around 360 Mbps in one direction, and 950 Mbps in the other, even with this Intel adapter.
local: 192.168.100.152 (win 8, realtek 8111c) remote: 192.168.100.154 (vista, intel desktop ct) 192.168.100.152 -> 192.168.100.154 113 MB/s 192.168.100.152 -> 192.168.100.154 104 MB/s 192.168.100.152 -> 192.168.100.154 103 MB/s 192.168.100.152 -> 192.168.100.154 104 MB/s 192.168.100.152 -> 192.168.100.154 102 MB/s 192.168.100.152 -> 192.168.100.154 104 MB/s 192.168.100.152 -> 192.168.100.154 101 MB/s 192.168.100.152 -> 192.168.100.154 102 MB/s 192.168.100.152 -> 192.168.100.154 101 MB/s 192.168.100.152 -> 192.168.100.154 104 MB/s ---------------------------------------------- Max: 113 MB/s Min: 101 MB/s Avg: 103.8 MB/s 192.168.100.152 <- 192.168.100.154 42.2 MB/s 192.168.100.152 <- 192.168.100.154 41.2 MB/s 192.168.100.152 <- 192.168.100.154 41.1 MB/s 192.168.100.152 <- 192.168.100.154 43.0 MB/s 192.168.100.152 <- 192.168.100.154 42.3 MB/s 192.168.100.152 <- 192.168.100.154 42.3 MB/s 192.168.100.152 <- 192.168.100.154 40.2 MB/s 192.168.100.152 <- 192.168.100.154 40.9 MB/s 192.168.100.152 <- 192.168.100.154 41.3 MB/s 192.168.100.152 <- 192.168.100.154 42.0 MB/s ----------------------------------------------- Max: 43.0 MB/s Min: 40.2 MB/s Avg: 41.65 MB/s
I have no idea why it peaked up at 113 MB/s in the first test run, local to remote. It kept that speed throughout the test run, the graph was almost flat at 113 MB/s. As before, I used a 60 seconds interval for each run. On the next run however it dropped to 104 MB/s.
As you can tell by these values, I still have the same throughput with this Intel adapter as with the built-in Realtek adapter. So I think it's safe to say that it has nothing to do with the adapter itself. So we can stop blaming RTL8111B as being an inferior/lesser chip than the RTL8111C found on the other motherboard. This looks more and more like a software/OS/configuration issue, or all three things at once.
Update 8 - Great results with Ubuntu LINUX
After exhausting all other options I finally decided to run some tests with Linux, and I got great results. I used a Ubuntu Linux 13.10 Live system and Iperf for Linux (version 2.0.5-3) on both local and remote machine. Here are the results.
======================================================= REALTEK 8111C <-> REALTEK 8111B | IPERF ON UBUNTU LINUX ======================================================= local: 192.168.100.152 remote: 192.168.100.151 192.168.100.152 -> 192.168.100.151 112 MB/s 192.168.100.152 -> 192.168.100.151 112 MB/s 192.168.100.152 -> 192.168.100.151 112 MB/s 192.168.100.152 -> 192.168.100.151 112 MB/s 192.168.100.152 -> 192.168.100.151 112 MB/s 192.168.100.152 -> 192.168.100.151 112 MB/s 192.168.100.152 -> 192.168.100.151 112 MB/s 192.168.100.152 -> 192.168.100.151 112 MB/s 192.168.100.152 -> 192.168.100.151 112 MB/s 192.168.100.152 -> 192.168.100.151 112 MB/s ---------------------------------------------- Max: 112 MB/s Min: 112 MB/s Avg: 112 MB/s 192.168.100.152 <- 192.168.100.151 110 MB/s 192.168.100.152 <- 192.168.100.151 111 MB/s 192.168.100.152 <- 192.168.100.151 111 MB/s 192.168.100.152 <- 192.168.100.151 111 MB/s 192.168.100.152 <- 192.168.100.151 111 MB/s 192.168.100.152 <- 192.168.100.151 110 MB/s 192.168.100.152 <- 192.168.100.151 111 MB/s 192.168.100.152 <- 192.168.100.151 111 MB/s 192.168.100.152 <- 192.168.100.151 111 MB/s 192.168.100.152 <- 192.168.100.151 111 MB/s ---------------------------------------------- Max: 111 MB/s Min: 110 MB/s Avg: 110.8 MB/s
Local (download, upload, idle)...
As you can see I get the same throughput in both directions when using Ubuntu. Is it because I use the same OS on both machines, or is this something else? Would I get the same throughput if I had the same Windows versions set up on both machines? I don't understand why it would matter if I use a slightly outdated Windows version, namely Vista, on one machine and the latest version on the other?... I mean Vista is still a current and supported OS and backed up by Microsoft. Windows XP is a different story.
But I know they are doing everything they can to kill off Vista. For instance, the latest Office 2013 is intentionally not supported on Windows Vista. I'm sure Microsoft wishes that Vista never happened. Just like they will be wishing that Windows 8.0 never had happened. But I'm usually just as persistent as they are, and I don't upgrade my Windows installs until I absolutely must.
So the question is how to get the same throughput in both directions with two different Windows versions. Windows Vista should be capable of Gigabit speed - it's not a 20 year old OS or something, it's not Windows 95 we're talking about. Vista is a modern OS. I haven't tested running the same Windows version on both machines yet. There might be a difference in TCP implementation or something between the two OS versions. If so, then I will probably be forced to upgrade the Vista machine. Either that or switch to Linux. I'm not prepared to pay more for less. Why would I have to upgrade Windows just to get Gigabit throughput in both directions?...
I tried reversing the cable. I got the same results as before. I also got a new Cat 6 patch cable and tried that one. The throughput test results were the same. So the cable is not the issue here. I only used pre-terminated/molded patch cables. So the wiring should be correct. But I plan to terminate my own installation cables later on.
FW and AV
As for firewall (FW) and anti-virus (AV), I don't use any third-party FW or AV software. I only have the Windows Firewall and the Security Essentials. I have disabled them both on both machines. The throughput test results were the same as before.
LAN Speed Test
I have installed LAN Speed Test Lite 1.3 on the local machine. I believe the test is performed between the memory on local and the disk drive on remote machine. I'm not sure. But it asks for a share path on the remote machine. I used o$ share on remote.
Upload: 427 Mbps Download: 420 Mbps
I don't trust these results too much. If you look at the graph you can see that it varies very much throughout the test. The test was a "successive" test, that is: write (upload) test first, and then read (download) test. Obviously if you do a simultaneous upload/download test the overall throughput will be lower. But I'm not interested in such tests. So far I have only been doing "successive" tests with both file transfer tests in Windows (file sharing/smb) and in Iperf.
I haven't done any memory to memory tests with LAN Speed Test because it requires the use of a program called LST Server on the remote, and this program requires registration in order to use it.
Disk drive tests
I used Crystal Disk Mark 3.0.3 to test the disk drives. Here are the results.
Local disk: 118 MB/s read, 113 MB/s write Remote disk: 70 MB/s read, 69 MB/s write
These are sequential read and write speeds based on 5 runs and a 1000 MB load.
This is the local disk (disk mark, read, write)...
And this is the remote disk...
But I don't get this... these results seem contradictory.
Okey, the local disk can read at 118 MB/s, so that would allow for the reported upload of about 100 MB/s. But the remote disk would not be able to receive it if it is only capable of 69 MB/s write. But by some magic twist I still get little over 100 MB/s upload on average.
Going the other way around makes more sense. If remote disk can read at 70 MB/s, and the local disk can write at 113 MB/s, then the download should not be any faster than 70 MB/s. I get about 40 MB/s download on average. That would seem reasonable.
So I can't conclude anything from these results. I mean the disk drive on the local computer is barely used. It is also the disk that holds the OS, and it is the only partition on that system. While the remote disk is almost full, and it is also partitioned with several partitions. However, it is not used for the OS. I chose drive letter
O: for the test here, because this is the partition with most free space.
(Note that I used drive letter
C: in the previous tests which is on a completely separate Seagate disk drive that holds the OS on the remote machine. So those readings are not comparable.)
With disk write caching enabled I got these results.
Local to remote: 106 MB/s Remote to local: 42.2 MB/s
I then disabled write caching on all drives on the remote and on the local drive.
I did not reboot because no reboot was requested for the changes to take effect. I then got following results.
Local to remote: 106 MB/s Remote to local: 42.1 MB/s
There was practically no change at all. No reboot, and no reboot was requested.
I then went on to disabling QOS Packet Scheduler for the appropriate adapter on the remote machine, and then on the local machine.
Local to remote: 107 MB/s Remote to local: 41.9 MB/s
No significant change here. Again, no reboot and no reboot was requested.
I then went on to enable jumbo packets I used the 4 GB setting because 4 KB is the largest MTU size that's supported on both machines.
Local to remote: 105 MB/s Remote to local: 33.3 MB/s
Now here, the upload (local to remote) was not affected, but the download throughput was reduced significantly. No reboot was requested, but I decided to reboot both machines anyway, just for good measure. I then performed the same tests again and got these results.
Local to remote: 117 MB/s Remote to local: 33.2 MB/s
So the upload is now even faster, but download is still slower than it was before I made these changes, even after the reboot. I would have expected them both to go up a little. What does this mean?