What is wrong with my fast-network's slow-network-share(s)?

Question

I have done extreme amounts of testing. cp, dd, rysnc, iperf, netcat, custom-dd and probably more.

What I am looking at is a strong network. The iperf results below show that, and I have other data to validate it as well.

What I am looking for is a sharing protocol that will match these speeds or come even close to them.

What I am currently getting with the sharing protocols iSCSI (FreeNAS) and CIFS (WIN7) is about 1.5 GB/m to 2.5 GB/m. My goal is 3.0 GB/m

What could I be screwing up in my sharing protocol implementation that is reducing the speed so drastically?

root@fdas:~# ./iperf -c 192.168.2.138
------------------------------------------------------------
Client connecting to 192.168.2.138, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.118 port 43066 connected with 192.168.2.138 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.05 GBytes   900 Mbits/sec
root@fdas:~# ./iperf -c 192.168.2.138 -t 60
------------------------------------------------------------
Client connecting to 192.168.2.138, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.118 port 43067 connected with 192.168.2.138 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  6.22 GBytes   891 Mbits/sec
root@fdas:~# ./iperf -c 192.168.2.138 -t 600
------------------------------------------------------------
Client connecting to 192.168.2.138, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.118 port 35506 connected with 192.168.2.138 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-600.0 sec  62.0 GBytes   887 Mbits/sec

//edit disk io benchmarks added. NOTE: this hdparm test reads in sequential order, and ultimate goal here is to be reading and writing raw bytes from a a disk (entire disk, not partition)

/dev/sdd1 on /a type ext4 (rw)

NOTE: sdd is FreeNAS iSCSI disk

root@fdas:~# hdparm -t /dev/sdd

/dev/sdd:
 Timing buffered disk reads:  168 MB in  3.02 seconds =  55.71 MB/sec
root@fdas:~# hdparm -t -T /dev/sdd

/dev/sdd:
 Timing cached reads:   5240 MB in  2.00 seconds = 2620.70 MB/sec
 Timing buffered disk reads:  168 MB in  3.01 seconds =  55.84 MB/sec

Answer 1

Have you run disk IO benchmarks on both the server and client? Unless the disk subsystems are capable of the throughput you require, they will of course be the bottleneck rather than the network.

Answer 2

In my experience iSCSI is the lowest-overhead of the bunch, an jumbo-frames do end up counting. I have seen iSCSI saturate a GigE connection using the LIO-Target iSCSI framework and a ramdisk as the target. That thing flew. The older version of the Linux iSCSI stack did have some performance issues in it, and couldn't use a ramdisk for full-bore throughput. I'm not sure what FreeNAS is running these days, the LIO-Target stuff is fairly recent.

One of the bigger limiters to such throughput ends up being the storage-system backend. As I mentioned, I got the above speed through use of a ram-disk (the server had 32GB of RAM, so it was worth trying). When I tried the same test using storage striped across 48 disks, I was able to saturate GigE during the sequential tests, but the random I/O tests were well below that; in the 65-80MB/s range as I recall.