up vote 3 down vote favorite
1
share [g+] share [fb]

After a few minutes of copying, it just gets slower and slower. Why?

e.g. It starts with 20 MByte/sec, and when it finishes with it, it's @10 MByte/s.

Various files, big, small, etc.

UPDATE: question is regarding various operating systems, so it's a "general question"

link|improve this question
62% accept rate
How much are you copying versus how much free memory do you have in your computer? – KCotreau Jul 26 '11 at 6:33
PC -> USB-FLASH drive: several hundred GBytes free -> few GBytes free – LanceBaynes Jul 26 '11 at 6:40
I suspect that you have files cached in memory at first so they transfer quickly, and then once they start being moved from the hard drive, it slows down. I can't be sure in your case though. – KCotreau Jul 26 '11 at 6:44
so you're advising me to use the "sync" command more often? – LanceBaynes Jul 26 '11 at 6:45
I don't understand what you mean by the sync command. You gave no context to the question. I am not even sure what OS. – KCotreau Jul 26 '11 at 6:54
show 1 more comment
feedback

1 Answer

up vote 3 down vote

This behaviour is not specific to your flash drive, you can see this with hard drives as well. It has to do with the caching mechanism most operating systems and disks employ to speed up small writes.

The 20mb/s you see is the data being written to the disk cache (usually quick but small memory). Once this cache is full it must be flushed to disk - and now you're being bottlenecked by the slower disk.

Example 1: The effect is really pronounced when you have a controller with a large cache (like good RAID5 controller) where ~500mb of data can be cached quickly before it must be flushed to the disk.

Example 2: You can see the cache at play if you pull the flash drive out at the same time the file copy "finished". At this point in time your file is split between the disk and the cache - so the copy is "finished" as far as the operating system is concerned but the disk controller still needs to write what's left in the cache to the disk. If you put the flash drive back in and inspect the file you'll see it's not all there.

Disclaimer: These examples won't work if you don't have write caching enabled in your operating system/on the disk.

Further, if it's not disk caching at work then it's likely that what you're seeing is an effect of fragmentation. As the disk gets fuller and fuller less contiguous free space is available and the file system has to work harder to find places to put your files.

link|improve this answer
Also keep in mind it is a flash drive. It's hardware was never optimized for speed, but rather affordability. – surfasb Jul 26 '11 at 7:45
I don't think that would explain why the performance would decrease during long file copies. Rather performance would decrease over the life of the drive, yes? – ta.speot.is Jul 26 '11 at 8:55
It makes perfect sense. They are optimized for burst speeds, not sustained operations. Think of the average size of files being loaded onto flash drives. Would you optimize for a 30 second long file transfer or optimize for a 2 second long transfer? Add that onto of your two examples, and it explains plenty. – surfasb Jul 26 '11 at 12:58
It's like when you try to cram two people through an average door at a time. At first, you can calculate. Wow, I went from zero people through that door to two at a time. My speed is great. Then the line gets longer and your sampling gets better and you finally realize that small sample = bad math. . . – surfasb Jul 26 '11 at 13:00
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.