Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I got my brain-melting first taste of Gigabit networking today, between my 2011 MacMini and Windows 8 Pro desktop connected via Cat.5e to Linksys WRT320N(sporting dd-WRT).

After making sure that the line speed on both systems showed 1Gbps, I proceeded to copying a 2.4GB MP4 from the Mini to the Win 8 desktop (SMB sharing). Although satisfied with the 30-34 MB/s that Teracopy was showing (that was a proper step-up for me from 10 MB/s), I still was curious about this massive difference in the advertised and real-world speed.

2 hours of Google had me believing that there were other factors that resulted in less speed, SMB being one. So just for the sake of doing it, I iPerf'd both the systems and guess what that showed - around 875mbps on both systems!

I then stumbled upon this little piece of info after which I turned off Teracopy and copied the same file through Windows 8's regular copier. 109 MB/s. Molten brains :)

What exactly is causing this? And can I enable such speeds via Teracopy? I really dig the extra features that Teracopy has, will surely miss them now :D

share|improve this question
1  
First thing which comes to mind is: How did you copy via teracopy? To/from a drive letter? (which serialises things and is slower) or to an UNC path ? –  Hennes Apr 13 '13 at 15:58
    
I've just shared the folders (Right-click>sharing>Share this folder, or equivalent)on both computers. I access the networked computer by simply opening "Network" from the left-hand pane in Explorer, and in OS X, by selecting the networked computer's name from the left-hand pane in Finder. To put it simply, I access the Mac's files from Explorer in Win 8, copy them using the basic Ctrl+C command, and paste it locally on the PC's desktop. –  Gaurang Apr 13 '13 at 19:04
    
What drives you use? Two SSDs? Those Windows 8 numbers look weird to me. –  Shiki Nov 10 '13 at 21:08

5 Answers 5

Over gigabit connection, Teracopy does 31MB/s, but Windows 8 does it at ~109MB per second?

What exactly is causing this? And can I enable such speeds via Teracopy? I really dig the extra features that Teracopy has, will surely miss them now :D

Two words: verification and cache

Technical Explanation

This is the general procedure for copying a file with Windows Explorer:

  1. Read a chunk of data from the source drive into memory
  2. Send the chunk through the system(s)
  3. Write the chunk to the destination drive
  4. If not done yet, return to step 1

This seems simple and short enough. With this transfer algorithm, each byte of the file is gets processed only two times: one read, one write.

But in addition, Windows uses memory (as does the drive itself) to cache some data. So instead of waiting for the previous chunk to finish getting written, and then reading the next chunk, a new chunk can be read while the previous one is still being written. Obviously this can’t hold up forever, but Windows can use up all remaining free memory as a temporary buffer to store most, if not all of the the file that is being read in memory.

You can see the caching in action by copying a large file—or a folder containing a lot of files—from one drive to another, then immediately comparing the two. The comparison will be much faster at that point than if you do it later on because the file(s) are still in memory, so it is not actually reading them from the drive(s).

Since memory is very fast, and read speeds tend to be a little faster than write speeds, the ultimate transfer rate ends up being limited only by the write speed of the destination drive.

Teracopy can do two things that can slow down a file transfer which Explorer does not do:

  • Forgo the cache and read directly from the drive

  • Verify that the destination was written correctly

Unlike Explorer which only checks for basic errors during the transfer, Teracopy can actually verify that the data was written correctly to the destination drive in order to prevent data corruption which could happen due to problems in the transfer media (network/drive cable/etc.) or the drive itself (bad sector, etc.) Doing this means that it has to read the file from the destination to compare it to the original.

Depending on the algorithm used and the size of the file, verification can be optimized as low as (but no less than) three drive operations for each file/file-chunk as opposed to Explorer’s two: read the file from the source, write the file to the destination, and read the file from the destination.

Look at what happens when you copy a file with Teracopy (with an HDD-optimized algorithm):

  1. Read a chunk from the source drive
  2. Hash the chunk read from the source drive
  3. Send the chunk through the system(s)
  4. Write it to the destination drive
  5. Clear caches
  6. Read the chunk back from the destination drive
  7. Hash the chunk from the destination drive
  8. Compare hashes
  9. Determine next step
    • If hashes don’t match, give error and prompt user for action
    • If hashes matched and not finished, go back to step 1

The problem is that if you cache the files during transfer, the comparison becomes useless because you are not reading the actual data on the destination drive, you are reading the copy cached in memory from the source. Therefore, to properly verify, you must clear the cache. This can be done after each read and write (which is an extra operation that would end up getting done countless times for files of any significant size), or just once after the whole file has been transfered.

According to the screenshot below, TeraCopy performs verification after the files are transferred, not during. This means that it uses this CPU/RAM-optimized transfer algorithm instead:

  1. Read a chunk from the source drive
  2. Send the chunk through the system(s)
  3. Write it to the destination drive
  4. If not done yet, return to step 1
  5. Copy finished, so clear caches and being verification
  6. Read a chunk from the source drive
  7. Hash the chunk from the destination drive
  8. Read a chunk from the destination drive
  9. Hash the chunk from the destination drive
  10. Compare hashes
  11. Determine next step
    • If hashes don’t match, give error and prompt user for action
    • If hashes matched and not finished, go back to step 6

While this algorithm puts slightly less strain on the CPU and RAM, it also puts a lot more strain on the drive(s) because now each file has to be processed four times: read the entire file from the source, write it to the destination, then read it again from the source, and again from the destination.

(If TeraCopy were optimized for network transfers, then it could avoid sending the whole file a second time for verification and send only the hashes which are much smaller, but that does not currently appear to be the case, and network drivers are treated the same as local drive, so it actually re-reads the source.)

By default, TeraCopy does not verify or use the cache. Not verifying would speed the transfer up (or more accurately, not slow it down), while not using cache would slow it down.

Application

To determine your specific speed results, you would have to check your settings to see if you have changed them. Then you can try to approximate what kind of speeds you would get with the different settings (be aware that they will likely interact, so it is not a simple matter of adding or subtracting).

That said, let’s do a cursory calculation using your numbers:

  • Local file transfer: 875Mbps (i.e., drive speed)
  • Network speed: 1Gbps = 119.21 MBps
  • Copy through Windows Explorer: 109MBps
  • Copy through TeraCopy: 34MBps

Right off the bat, we see that Explorer’s file-transfer is nearly maxing out your network throughput. A 1Gbps link is equal to 1,000,000,000 bits per second, which in binary units of bytes, is 119MBps. Explorer is clocking 109MBps, and the remaining 10MBps (which amusingly enough was your previous max :^Þ) can easily be accounted for by overhead, background load, and fragmentation.

(Since the transfer rate is almost equal to the network speed, we can surmise that Explorer’s file transfer is one-way and only a single copy of each file is getting sent.) Explorer gets 110MBps for two file accesses per file.

Now for TeraCopy. It seems that TeraCopy is getting almost exactly one-third of Explorer’s speed.

Depending on whether or not its preferences dialog is accurately indicating the algorithm it uses, then TeraCopy could actually be sending the entire source file twice so that it can check the copy. Right away, this cuts the throughput in half. If the cache is on, then that too reduces the speed because it has to wait for each write to finish before it can send a new chunk. When combined with verification, it can knock it down even more.

Your transfer rate of 34MBps seems reasonable if you have verification on and caching off. If you turn verification off and caching on, you should get about the same as Explorer (you may still get a slightly less or even slightly more depending on just how different the file-transfer code that TeraCopy uses is from that which Explorer uses).

If you’re in the mood to transfer nearly 10GB, then you could also just try altering the settings and redoing the transfer for each of the four permutations and note down the speeds you get (to be safe, order it so that the cache is off between runs: V+C+, V+C-, V-C+, V-C-).


Screenshot of TeraCopy preferences dialog with default settings

share|improve this answer

I cannot give you a 100% answer to your issue but I can add more that might help.
My reason for switching to Teracopy was due to the way Windows has to "calculate" everything before doing anything and this calculation itself can sometimes freeze the whole process. In other posts it was brought up that the type of file being copied mattered in Windows copy, I cannot say for sure in Teracopy as I have not seen that brought up. But my own attempts have so far left me with less than you get in any event. I am just copying Generic file types, not media( ie. photo/movie) files.

My tests went from a Seagate 3TB NAS with GB NIC access on a 100% GB network to a GB system. Best case, 14 Gbps. Windows move from NAS to system HD got 12.4GBps and took slightly longer to start. Nothing approached your 30-34Mbps much less anything like 109Mb/s.

Part of this is surely due to the NAS's flow control even on a GB network. I would be extremely happy if I could get 34GB/s, happier still if I could get 109 GB/s. If you find a faster file copier please post! :)

share|improve this answer
    
I would like to add that the setting for using system write cache when used does appear to make a significant difference in real-time completion. Much Faster. –  user1376156 Jul 9 '13 at 3:04

I would suspect it's related to the fact that Windows 8 uses SMB v. 3 and I doubt teracopy is compatible with SMB v.3 and is defaulting back to SMB 2. Just a guess...

share|improve this answer

This helped me with this issue, between Intel motherboard chipset NICs.

The TOE (TCP Offload Engine) was introduced to Windows in 2003, with horrible results from the Broadcom NICs used in most servers at the time causing data corruption if you "offloaded" the checksumming. It looks like the NIC vendors still don't have it right, and it isn't just Broadcom. The network offloading is almost always slower and apparently also more error prone then doing it all in software. I would go in your adapter settings and disable all hardware offloading of any sort.

share|improve this answer
1  
If this was the cause then all TCP network traffic would be slow. Not just teracopies network yes a fast explorer copy. –  Hennes Dec 25 '13 at 9:13

I did also several test with Teracopy 2.3 final and my Synology Diskstation. In fact the copy process is too slow. I´ve copied files with an average Speed of about 40 MB/s.

The original Windows 8 file copy program is faster than Teracopy!

Now, I'm currently giving Ultracopier a try, my current speeds are about around 80 MB/s, much faster than Teracopy. I like Teracopy, but I think Teracopy has a network copy problem.

Because I'm copying many files from one network drive to another I must use the fastest copy program I can get, because the copy process always is too long.

share|improve this answer
    
I was just about to try Ultracopier... And I google searched it and wiki states that it is malware disguised as a replacement for windows explorer file... CAUTION! BEWARE! "Ultracopier is malware that disguises itself as a tool to replace the file copying functionality of Windows Explorer/File Explorer." en.wikipedia.org/wiki/Ultracopier –  Amean Dec 9 '14 at 15:30
1  
Ultracopier is a malware - I don't recommend using it. (en.wikipedia.org/wiki/Ultracopier) –  smc Dec 9 '14 at 19:39
1  
If you actually take the time to read the article linked by that wiki page it is disclosed: "To fund it, I have put a Bitcoin miner (which uses the GPU at 100%), or version without it but with a small advertisement into the free versions." But it is a valid point about the software not clearly stating such on it's download site. –  Mokubai Dec 9 '14 at 19:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.