27

Robocopy has a /J command line option recommended for copying large files (it copies using unbuffered I/O).

What (if any) downsides are there? Any reason this isn't enabled by default? (That's what made me think there MIGHT be downsides.)

3
  • I can imagine some performace downsides with lots of small files. But with large files? Not many. It might be slower. I expect it to be much more predictable when copying to a slow destination. Lets see which answers we get from other users since I am just guessing atm:)
    – Hennes
    Aug 16, 2016 at 21:03
  • Hmmm... if it might be slower even on large files, I wonder what the BENEFITS are then. I updated the question to reflect that. Aug 16, 2016 at 21:07
  • When I copy (using windows explorer, not robocopy) from fast HDD to external (slow) USB drive on a system with 18GB RAM (read: lots of memory which can be used as disk buffer) I often ran into situations where reading source files was done, yet unmounting the slow USB2 disk took about 45 minutes while cache was being flushed. I wish I could have limited cache memory there. This might just be the option for that in robocopy.. ANyway, post tagged, it will be interesting for both of us to see which answers show up.
    – Hennes
    Aug 16, 2016 at 21:10

3 Answers 3

20

Great question.

Unbuffered I/O is a simple file copy from a source location to a destination location. Buffered I/O augments the simple copy to optimize for future reads of (and writes to) the same file by copying the file into the filesystem cache, which is a region of virtual memory. Buffered I/O incurs a performance penalty the first time the file is accessed because it has to copy the file into memory; however, because memory access is faster than disk access, subsequent file access should be faster. The operating system takes care of synchronizing file writes back to disk, and reads can be pulled directly from memory.

The usage note mentions large files vis-à-vis buffered I/O because:

  1. The up-front cost is expensive. The performance penalty with buffered I/O is substantially worse for large files.
  2. You get little in return. Large file blocks don't tend to stay in the cache for very long anyway, unless you have a ton of memory relative to the file size.
  3. It may not avoid disk I/O. Reads and write of large file data blocks increase the probability of requiring disk I/O.
  4. You probably don't need to buffer anyway. Large files tend to be less frequently accessed in practice than smaller files.

So there is a tradeoff, but which is appropriate for you depends on your particular case. If you are zipping up a bunch of files and transmitting the zip to a backup target, unbuffered is the way to go. Copying a bunch of files that were just changed? Buffered should be faster.

Finally, note that file size is not the only factor in deciding between buffered and unbuffered. As with any cache, the filesystem cache is faster but smaller than the source behind it. It requires a cache replacement strategy that governs when to evict items to make room for new items. It loses its benefit when frequently-accessed items get evicted. For example, if you are synchronizing user home directories intraday to a separate location (i.e., while users are actively using the files), buffered I/O would benefit from files already resident in the cache, but may temporarily pollute the cache with stale files; on the other hand, unbuffered would forego any benefit of files already cached. No clear winner in such a case.

Note: this also applies to xcopy /J

See Microsoft's Ask The Performance Team Blog for more.

3
  • "Copying a bunch of files that were just changed? Buffered should be faster." <-- Why is that? Would the cache have been pre-existing in the memory for Robocopy to take advantage of? Shouldn't Robocopy need to be called first before future I/O request on the same file to get the benefit? Robocopy does potentially write new data after all.
    – Argyll
    Jan 9, 2023 at 4:18
  • @Argyll the filesystem cache exists independently of processes that use it. The idea is that actively used files are more likely to be in the cache already before Robocopy is called. There's a link in the answer to an MSDN article that describes file caching in detail. Hope this helps. Jan 11, 2023 at 1:49
  • I see. Thank your the explanation
    – Argyll
    Jan 12, 2023 at 4:52
4

If you are copying across the WAN, I recommend NOT having the /J option enabled for large files as your average copy time will increase significantly. The files I copied were anywhere from 500MB to 23GB.

On a 50Mbps line, I averaged 43.5Mbps (other traffic and overhead) while never going below 32Mbps WITHOUT /J. With /J my average was around 25Mbps...looking at perfmon, I could see large peaks and valleys at the bottom.

Hope this helps someone out.

3

I tried the following:

When you copy from a fast device (NAS via Gigabit-Ethernet) to another fast device (USB3-Disk)

  • without /J: the data is read into a buffer and written after that, so either the network or the harddrive is idle
  • with /J: the data is read and written without wait, so the network and the hard drive are used simultanously

I would suggest to use this option.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .