Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I use Debian 6 on a HP Elitbook 6930 with 2Gigs of RAM. I was copying two AVI files, 1.5 GB in total, and noticed that the data copying was done at the rate of 4MB/sec. When I copy same AVIs to my Western Digital Passport 25G USB plugin drive the data transfer speed is 11+MB/sec. This speed is different if I plugin the drive to different USB ports. Interestingly, at work I also downloaded a 16MB IE8 exe on my virtual xp, run inside Oracle Sun Virtual Box, and it was downloaded AND saved within a second. We have a high speed network at work. :-)

Why and how this difference in data speeds is possible?

share|improve this question

migrated from stackoverflow.com Feb 25 '11 at 5:55

This question came from our site for professional and enthusiast programmers.

2 Answers 2

up vote 1 down vote accepted

There is a lot of factors influencing speed of file transfers. I recommend you read the following wiki articles to get a better understand of what is happening when a file is transfered from one device across you computers busses / (in-put/out-put) hardware to an other device. This answer is mostly theoretical. The explanation and examples are there to help you understand some very complicated concepts and are not 100% technically correct.

  1. Computer Bus
  2. Hard Drives
  3. File System
  4. Universal Memory(very interesting look at the future of storage)

Understanding buses

The computer bus is the actual wires over which the data moves. Different buses like Universal Serial Bus (usb) and Ethernet transfer data using different protocols, through different controller chips or integrated circuit (IC). On different computers they will use different IC in different configurations, which will affect the transfer speed. Some cheaper systems will rely on the CPU while others will have a dedicated controller just for handling data transfers. USB on most computers now will have it own controller chip, and will be able to off load allot of processing work from the CPU. USB may have more then one device using the same communication channel. For example your mouse and hard drive can share the same set of wires to the controller circuit which means that every time you move your mouse you interrupted the hard drive.

In order to appear to speed things up most buses have logic that allows them to use (direct memory interface) DMI. Which means that bus will be given a portion of RAM in which to buffer / cache data. So your 16MB download coming across the network can be buffered before it is actually written to the hard disk. Where as the large 1.5 GB file transfered will fill up the buffer. So the first part of the transfer will be quite fast till the buffer is full, after which it will revert to the actual transfer speed of the device. You will now see disk advertising there buffer size. Like 32mb and so on. So while USB2 advertises a speed of 480mb/s transfer rate the buffer of the disk is on 32mb once it is full we are back to the actual transfer speed of the device.

Understanding Hard Drives

Hard drives are mechanical devices the are affected by the speed at which the device can actually move to a portion of the disk platter. The wiki atrical on hard drive speed will explain this quite well. The disk arm / head has to move from its resting position to over where the file data is actually located and then read or write depending on what operation you were performing. Read are faster as there is less work done in detecting the state of a portion of the disk, then when actually changing that portion. The more fragmented a file is, ie the more spread out over the disk the longer it takes for the mechanical components to move into place to do a read or write. You will notice that the advertised bus speed of ESATA is at about 3GB/s. You disk mechanical and circit logic are two sepearte parts. So while the disk buffer of say 32MB will be filled at teh 3GB/s stat bus speed. The mechanical components take much longer to do there job, which is where even on the most high end mechanical disks you will still only transfer at between 15mb/s - 30mb/s.

Understanding File Systems.

Now the file system on an operating system as apposed to that of a disk is a little more complicated. You need to take into account the virtual memory of an operating system to understand how a file is created, read and written to. Which is one of the reasons you need to unmount/eject a disk before you pull it out as some file may not have finished transferring and have continued to do so in the back ground. If you do it before it is completely finished you can corrupt the file you were transferring or worse the file system. Virtual memory is essentially a huge cache / buffer used by the operating system to keep the most recently used data in quickly accessible RAM.

The disks file system such as NTFS / ext#(1,2,3,4) are how the operating system is able to turn a computer understandable address such as CHS (Cylinder-head-sector) into human understandable file name or data. This requires work by the operating system in order to see which CHSs' are available and where your data is. This is where fragmentation come in. If you have many free regions spread out over the disk it will take a long time for the mechanical components of the disk to find them and do a write operation, as writes take longer then reads.

Typically during a transfer you are limited buy the transfer rate of the writing device, and not the transfer rate of the reading device. Larger files take longer as more free space will need to be found and there is a good chance that not all of it will be contiguous (not fragmented). Now if you are moving files from one partition to an other on the same disk this is the worst case scenario as far as transfers are concerned. Think about the disk arm moving to one address to read and then to another to write, once the buffer is full. This will really slow down your transfer.

Conclusion

I can't see exactly how your system is set up but I hope you are able to see how much is involved in the simple act of copy and paste. The two biggest factors affecting the speed of reads and writes are the mechanics of your disks and the file systems state. NTFS / FAT 32 will be sower when not defraged. Linux(ext3,4) will does have defrag software that can help, but tends to be much better at allocation contiguous regions then windows. Linux ext# is still not perfect and you could improve speed by defraging but I doubt that you will actually notice a difference.

share|improve this answer
    
Thanks a ton for this explanation. –  Kumar Feb 28 '11 at 4:28

"The mechanical components take much longer to do there job, which is where even on the most high end mechanical disks you will still only transfer at between 15mb/s - 30mb/s."

Above statement is incorrect! Modern desktop/laptop hard drives have a sustained sequential transfer rate above 100MB/s (Megabytes per second). Even 5400RPM drives now days have a transfer rate above 100MB/s.

Random seeking and transferring data that is held or is to be written to random places on the hard drive will be much slower. 15-30MB/s indeed, but usually a bit faster still.

In a few years, as data densities increase further, you will see most new desktop/laptop drive capable of over 200MB/s sustained sequential transfer rate speed.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.