Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a small media server at home which has software RAID and a gigabit link to the rest of my network.

For some reason though, I only get ~10MB/s transfers when copying to/from the server.

I use software RAID5 (mdadm) over 4 1TB disks. On top of that I then use LVM to give me a huge pool of disk space which is then split up into multiple partitions which can be resized as and when they need it. I'm guessing this it most likely the cause, but I'd like to know for sure where the root cause is.

So, how can I benchmark network throughput (Windows 7 desktop <-> Ubuntu server) and hard disk performance to try and identify where my bottleneck might be?

[Edit] If anyone's interested, the motherboard is an Intel Desktop Board D945GCLF2. So that's a 300 series Atom processor with the Intel® 945GC Express Chipset

[Edit2] I feel like such a fool! I just checked my desktop and I had the slower of the two onboard NICs plugged in so the server is probably not at fault here. Transferring a copy of ubuntu off the server I get ~35-40MB/s according to Windows 7. I'll do those HD tests when I get a chance though (just for completeness).

share|improve this question

migrated from serverfault.com Apr 10 '10 at 6:51

This question came from our site for system and network administrators.

4  
Software RAID with an Atom CPU... Don't look any further. Especially RAID5 which does parity calculation. Also I suppose your network is 100Mbps which gives you a max throughput of 12Mbps, so 10MB/s is good =) –  Antoine Benkemoun Apr 9 '10 at 9:13
    
Jon has a Gigabit Link. –  lajuette Apr 9 '10 at 9:25
    
@lajuette - having gbit components is not always enough, you need to verify that each link in the chain has negotiated a gbit connection and not fallen back to a slower link speed due to odd incompatibilities, cable faults, or random happenstance. –  David Spillett Apr 9 '10 at 10:50
    
This isn't an answer but rather recommending a fix - The simplest solution seems to be to add a hardware-based RAID card. It'll offload IO overheads from your limited CPU, and if you get one with a reasonable cache (you should!), you'll see additional benefits from that. Seems to be your easiest plug-and-go option at this point. Additionally you will benefit from enabling Jumbo Frames, if all your kit supports it. –  Chris Thorpe Apr 9 '10 at 11:05

4 Answers 4

Like Antoine said: An Atom CPU and SW RAID is a bad idea. To measure the troughput of your disks you can use hdparm.

Have a look at this: http://www.cyberciti.biz/tips/how-fast-is-linux-sata-hard-disk.html

You should measure your disk devices and your raid devices seperately. That way you can see if the disks are slow (broken?) or the RAID is slow. Also have a look at your cpu usage (e.g. with top) while measuring or accessing your RAID ome other way.

If that's not the bottleneck check if your GBEthernet Link is using its full capacity. Have a look aht the output of ifconfig. Mine reads as follows (Mac OS X 10.6, it should look similar on Ubuntu):

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    ether 64:b9:e8:bf:8f:b4 
    inet 192.168.0.5 netmask 0xffffff00 broadcast 192.168.0.255
    media: autoselect (1000baseT <full-duplex,flow-control>)
    status: active

2nd line from the bottom: 1000baseT means: GB Ethernet!

[edit] I found this article: http://www.performancewiki.com/diskio-monitoring.html It recommends sar and iostat for monitoring disk IO.

share|improve this answer
    
Sounds like good advice - I'll give your suggestions a try. –  Jon Cage Apr 9 '10 at 9:35
    
I'm interested in your results. If the Atom (which one btw.?) is capable of Doing such stuff i want to try this at home. I'm running a media center on a ION board. –  lajuette Apr 9 '10 at 9:36
    
@lajuette - an Atom should do well as a basic file server if that is what you are meaning. I've not tested one with a Gbit NIC but as Jon's results show it is capable of showing 10Mbit speeds with a RAID5 array, and in theory should be capable of significantly faster. I once did some tests on my netbook to see how an Atom would do as a VPN server too and it could push/pull ~85-90% of a 100Mbit link's capability when using OpenVPN as the secure transport layer, you ION board probably has a more capable chip from the Atom family so may fair better still. –  David Spillett Apr 9 '10 at 10:47

A couple of things to check:

  • make sure the NIC is actually in Gbit more, not 10Mbit with ethtool eth0 (replace eth0 with the relevant NIC's device ID if different), look for the "speed" reading for the current mode
  • verify that Windows is currently using its card in Gbit mode too
  • check to see what CPU and I/O load is imposed on the server during the transfer - if you see (in top or similar) one core at near 100% I/O-wait state then your drives are the bottleneck, if you see high "system" CPU use then the CPU it the bottlneck (it could be a mix of the two for writes due to RAID 5's "write -> read+read+write+write" performance hit on four drives and the parity calcs being done by the CPU, if neither system or IO readings are high then the network is most likely the bottleneck.
  • test bulk read performance purely on the server (to test raw IO performance irrespective of the network): cat a large file to /dev/null to see how far that goes (do echo 3 > /proc/sys/vm/drop_caches first so you know the IO is actually hitting the disks and not coming from memory, and if you have it installed use pv instead of cat as it gives useful speed+progress readings), watching the CPU+IO load as this happens too
  • test bulk write performance with cat /dev/zero > /some/file/on/the/array (or pv /dev/zero > /some/file/on/the/array), watching the CPU use as this happens too.
  • test bluk network throughput, irrespective of the drive/array performance, between the machines with netcat and pv - on the Win7 machine do nc -l -p 123 > NUL and on the server then do pv /dev/zero | nc 1.2.3.4 123 where 1.2.3.4 is the Windows box's address (you may end to add a firewall exception for nc).

As you are seeing a flat rate of ~10Mbyte/sec I would suspect a network issue first rather than a bottleneck at the disks or CPU - but RAID5 on an Atom CPU might be one of the bottlenecks so you might want to consider RAID1+0 or RAID10 instead (if your RAID5 array is 3-plus-spare then Linux's RAID10 driver with 3+spare should give similar redundancy (any single drive failure is survivable) but with better performance (write->write*2 rather than write->read+paritycalc+write*2), in 3-drive mode the RAID10 driver does something similar to what IBM controllers call RAID-1E, see http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10).

edit:

Added extra thing to test to the list above, and other minor detail

share|improve this answer
    
The ~10MByte seemed suspicously close to a 100MBit link to me too. I'll give your suggestions a try when I get a chance - thanks! –  Jon Cage Apr 9 '10 at 9:34

First of all, you are using SMB/CIFS, which is not a very fast protocol, (it is definitely slower that NFS, for reference).

Second, it depends what workload you are testing. Is it mostly sequential or random? If it is mostly random I/O, then 10MB/s is probably OK. Realistically, from a GB NIC you can expect 30-50MB/s from CIFS (but, as I said, could be higher or lower, depending on the workload).

You can also check this other answer from serverfault for CIFS performance tuning.

A quick search revealed this page with CIFS performance bechmarks. You may find it useful.

Finally, you can test the performance of the network with iperf (it can be compiled for windows as well, and you may find it precompiled somewhere)

share|improve this answer
    
I was transferring large data files so big sequential writes / reads. –  Jon Cage Apr 9 '10 at 10:07
    
@Jon: Other than advising you to test the network itself and to check the CPU load on the fileserver, I can only direct you to the official Samba performance page samba.org/samba/docs/man/Samba-HOWTO-Collection/speed.html –  Dan Andreatta Apr 9 '10 at 10:51
up vote 0 down vote accepted

Turns out it was the desktop machine I was using; it was running at 100MBit. Thanks for all the advice though - could be very useful for benchmarking and improving the overall speed of my system!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.