Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have done extreme amounts of testing. cp, dd, rysnc, iperf, netcat, custom-dd and probably more.

What I am looking at is a strong network. The iperf results below show that, and I have other data to validate it as well.

What I am looking for is a sharing protocol that will match these speeds or come even close to them.

What I am currently getting with the sharing protocols iSCSI (FreeNAS) and CIFS (WIN7) is about 1.5 GB/m to 2.5 GB/m. My goal is 3.0 GB/m

What could I be screwing up in my sharing protocol implementation that is reducing the speed so drastically?

root@fdas:~# ./iperf -c 192.168.2.138
------------------------------------------------------------
Client connecting to 192.168.2.138, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.118 port 43066 connected with 192.168.2.138 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.05 GBytes   900 Mbits/sec
root@fdas:~# ./iperf -c 192.168.2.138 -t 60
------------------------------------------------------------
Client connecting to 192.168.2.138, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.118 port 43067 connected with 192.168.2.138 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  6.22 GBytes   891 Mbits/sec
root@fdas:~# ./iperf -c 192.168.2.138 -t 600
------------------------------------------------------------
Client connecting to 192.168.2.138, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.118 port 35506 connected with 192.168.2.138 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-600.0 sec  62.0 GBytes   887 Mbits/sec

//edit disk io benchmarks added. NOTE: this hdparm test reads in sequential order, and ultimate goal here is to be reading and writing raw bytes from a a disk (entire disk, not partition)

/dev/sdd1 on /a type ext4 (rw)

NOTE: sdd is FreeNAS iSCSI disk

root@fdas:~# hdparm -t /dev/sdd

/dev/sdd:
 Timing buffered disk reads:  168 MB in  3.02 seconds =  55.71 MB/sec
root@fdas:~# hdparm -t -T /dev/sdd

/dev/sdd:
 Timing cached reads:   5240 MB in  2.00 seconds = 2620.70 MB/sec
 Timing buffered disk reads:  168 MB in  3.01 seconds =  55.84 MB/sec
share|improve this question
add comment

2 Answers

up vote 2 down vote accepted

Have you run disk IO benchmarks on both the server and client? Unless the disk subsystems are capable of the throughput you require, they will of course be the bottleneck rather than the network.

share|improve this answer
    
I have added minimal hdparm -t results above. Do you have any benchmarking suggestions? –  c card Dec 15 '11 at 16:26
    
Common disk IO benchmarks are bonnie++, iozone, fio. Make sure the size of your test file(s) are at least 2*RAM. –  janneb Dec 16 '11 at 6:12
    
After running hdparm and bonnie, the results showed the disks were the problem as you suggested. I upgraded the source disk to a solid state drive, and the destination disk to two disks in a raid0, and the results were about 6 GB/m -- THANKS –  c card Dec 19 '11 at 14:21
add comment

In my experience iSCSI is the lowest-overhead of the bunch, an jumbo-frames do end up counting. I have seen iSCSI saturate a GigE connection using the LIO-Target iSCSI framework and a ramdisk as the target. That thing flew. The older version of the Linux iSCSI stack did have some performance issues in it, and couldn't use a ramdisk for full-bore throughput. I'm not sure what FreeNAS is running these days, the LIO-Target stuff is fairly recent.

One of the bigger limiters to such throughput ends up being the storage-system backend. As I mentioned, I got the above speed through use of a ram-disk (the server had 32GB of RAM, so it was worth trying). When I tried the same test using storage striped across 48 disks, I was able to saturate GigE during the sequential tests, but the random I/O tests were well below that; in the 65-80MB/s range as I recall.

share|improve this answer
    
Thank you for your help –  c card Dec 19 '11 at 14:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.