Will robocopy resume after being aborted?

Question

My users directory is 58 GB in size and I am copying it to a portable USB backup drive from Windows recovery using Robocopy.

robocopy G:\Users\ E:\Backup\T420\ /MIR /SEC /MT:32 /V /LOG:G:\bak\robocopy.txt

The laptop is USB 2 and the backup drive is USB 3.

It has been running for over 18 hours now which is longer than my expectation.

Even at a speed slow for USB 2 it should only take about 8 hours

https://www.wolframalpha.com/input/?i=58+gB+at+2+MB%2Fs

If I press Ctrl-C and inspect the log to see what is going on then run the command again will it take the same amount of time or will it resume from where it left off?

Assuming it resumes, what sort of time overhead is there to work out which files are left to copy and where to resume from?

EDIT: It turns out that the reason it was taking so long was because of some fake/not-fully-sync'd OneDrive files that couldn't be copied and default values of retrying 1 million times. Changing the retry and wait allowed it to complete /r:2 /w:5

Answer 1

If the folder contains a very large number of files, then it will indeed take a long time, required by the frequent accesses to the disk catalogs, for both source and target.

In addition, a USB3 disk on a USB2 port will work at the speed of USB2. The USB3 disk will adapt itself to the port and its protocol when doing the initial handshake.

You are using /MT:32 for Multi-Threaded, so telling robocopy to copy 32 files at once. This will cause 32 threads to fight for access to the disks, and will slow the copy even further, with the disk's head wasting time by going back and forth over the disk. In the case of slow USB2 copy, Multi-Threaded is not recommended.

Use Restartable mode /Z to process partially-copied files. With this option, when the copy is interrupted so some particular file is partially copied, the next execution of robocopy can pick up where it left off rather than re-copying the entire file. There are unfortunately cases where this switch will slow down the copy.

Answer 2

It is OK to stop Robocopy in the middle of an operation.

By default it will not re-copy the files that have already been copied.

The overhead for determining where to resume from (or essentially to check which files are the same) is very low - only seconds.

However, if you are not writing to a log (or are using /TEE to also write to a console) the /V will increase this overhead as it will take longer to write all the lines to the console.

NB: Restartable mode (as mentioned in the comments) is just for restarting individual files (e.g. if a giant file is stopped in the middle you can resume part way through instead of copying the whole file again) and is not required for resuming batch copying as asked in the question.

Answer 3

I've used robocopy for years now, and know that you can stop a copy in the middle of a file and restarting will swiftly get back to where it left off. Unless you use the /XN option, robocopy will delete whatever the last file it was working on and start the copy again. When it first creates a file, it sets the "Modified" time to 1/1/1980, so basically as long as your clock is set correctly, it will see that time and restart the copy process. As you've stated, other files will be ignored since they have the same modified date/time as the source files.

Personally, I always use the /TEE option along with the /V and /LOG: options when using robocopy so that I can monitor what is going on, (and if a crash happens, I can go back and see if there were issues as well). It will slow things down a little bit if you have an extreme amount of small files (tens of thousands or more), but it can also help you determine if there's a problem. If you shrink the height, or better yet, minimize the command prompt window, you can greatly speed up the copy when you have a bunch of small files.

18 hours is very long for the amount you're trying to copy, so it's possible that you either have an insane amount of small files that you're trying to copy, or you have some recursion in your directory structure. Since you have a log, you could try looking at the bottom of it and see if the directory structure is repeating itself. If it is, you could try skipping junction points with /XJD to see if that allows you to stop the loop(s) it's getting stuck in. The /XD option can also be used to completely ignore specific directories.

About the /Z option, this option ONLY works for "network transfers". I put that in quotes because local copies will NOT use the restartable mode; however if robocopy sees a network path, it will use restartable mode, even if the source and destination are local, (this could just be my version of robocopy since I'm not moving past Windows 7). If you share out your drive or a folder on the drive, (preferably to an account that only you have access to), and enable the Change permission(s), then use that path as the destination, it should work in restartable mode. Granted, there may be a hit in performance when using restartable mode.

Answer 4

Is it OK to stop Robocopy?

I kill the Robocopy task often without a problem. If the process is killed, the file that was being copied will, as far as I remember, have (a) time stamp of 1980-01-01 (b) size same as the original file.

Assuming it resumes, what sort of time overhead is there to work out which files are left to copy and where to resume from?

If you start the task again with those parameters, it will (a) skip the files that it copied already (b) re-copy the file that was interrupted because of date mismatch (c) copy the files that it did not in previous attempt (d) delete the files from destination that no longer exist in source because of /MIR switch.

It turns out that the reason it was taking so long was because...

Before killing the process, you can open the log file in a text editor to see what it is doing; the last line will contain the file it is copying and its size. If there was an error, that too will be listed such as:

2019/12/13 06:26:38 ERROR 32 (0x00000020) Copying File C:\Redacted
The process cannot access the file because it is being used by another process.

Waiting 60 seconds... Retrying...

Once the copy is successful, the log file will contain a summary that includes the transfer rate (megabytes per minute) that will give you a rough idea of how much time it'll take next time.

Answer 5

I'm going to premise-challenge a bit here. I understand the frustration of a slow copy, but interrupting the copy is not how you check copy progress.

Look at the disk I/O history chart

Fire up Performance Monitor and look at disk I/O activity history. You should do this with the system quiescent; if it's doing a lot of other stuff you could be looking at that disk activity. However, that will be spiky; you're looking for the "baseline" activity that is there all the way across the chart. Mine on the Mac splits it into reads and writes; obviously you're looking at writes. (most other activity will be reads).

Older Mac version. Windows' is more robust. The first lump is rsync planning the copy, the second lump is three fat MP4 video files, and the last lump is a bazillion little files.

In my experience, In practice, this bar will jump up and down quite a bit. When copying large single files - videos and TIFF files, it pegs at whatever the hardware maximum throughput is.*

But when copying bazillions of little files, MBps throughput tanks ... because the drives must read (and write) a directory entry for each file. You don't have head seek times and wait-for-sector-to-come-around delays on Flash memory keyfobs, but you do on physical hard drives - bigtime. Hard drives and robust OS's have some strategies to ease the pain, but there's still big pain.

• This is why I turn all my collections of little files into ZIP packages, tarballs, or disk images.

Look at the files themselves arriving on the destination drive

Use your windows (or a command prompt) to browse the destination directories. Find the "leading edge" of the copy, i.e. the directories being added into right now. Watch the activity and see if it's normal.

Listen to the hard drive

If your drive is going

tick... tick... tick...

That's a copy going well. The drive is doing big block transfers, using its formatted sector arrangement to best advantage, etc. If on the other hand you here

shikkita shikkita shikkita

That's disk thrash. Something has gone wrong with your backup, and it's killing throughput. That sound means the drive is seeking the head a lot - and when the head's moving, it's not copying. Worse, when it arrives, it must stabilize and then wait an average of 0.5 disk platter revolutions for the desired sector to come around. Drives spin at 5400-10,000 RPM, so that's 3-6 ms every seek.

In the earlier diagram, indeed, the fat section had the drive quiet as a mouse, and in the later section, shickkita.

For instance, once I had a nest of 5.5 million tiny (30-500 byte) files in the middle of a big backup. When the drive started sounding like a World War I melee, I checked the graph and saw the bad news. At that rate, the copy would've taken days.

So I quit apps, created an ample RAMdisk, and copied the 5.5M files to the RAMdisk. Reading from HDD, this benefited from disk cache; writing was instant of course. This took a half hour. Then I wrote it back to hard drive as a ZIP file, which wrote as one continuous data stream, so "tick...tick...tick...". That was even faster.

Deleting the 5.5M files took an age, but then, the ZIP file backed up in less than a minute. Big improvement over a day!

As far as when to interrupt the copy, you do that when the above is telling you the copy is not working properly, and you have a plan or want to experiment to fix it.

Never, ever, ever multi-thread

The meters show you'll get best throughput when streaming a single large file like a video. The drive's sector optimization is working at peak performance: the next needed sector appears under the disk head just as it's needed.

But suppose you copy 2 large videos at the same time. The disk is trying to do both at once: It seeks to video 1, writes a block, seeks to video 2, writes a block, seeks to video 1, writes a block, ad nauseum. Suddenly we're seeking instead of writing, and throughput does the obvious thing.

So don't do that. And the disk I/O history chart will tell you why.

For instance I "rsync" from internal HDD to External 2. I also rsync from External 1 to External 2. These are all set up in scripts. I don't run both at once, because they'll fight over External 2, and it'll slow both copies down.

Now what you did, with /MT:32, is tell it to thread 32 copies at once, which is the very thing I'm here telling you not to do. *Maybe multi-threading helps on hybrid, RAM disks, RAIDs, or good elevator-seeking algorithms that play nice with the particular data-set. That doesn't sound like it's here. But regardless, don't take my word on it -- you should be using the graphs to experiment and find the happy number for your hardware. (Which I fully expect will be "1").

---

* lowest of a) computer USB, b) device USB, c) read drive throughput and d) write drive throughput.

Answer 6

This answer addresses:

It has been running for over 18 hours now which is longer than my expectation.
Even at a speed slow for USB 2 it should only take about 8 hours

and suggests a possible alternative to Robocopy

---

TRANSFER SPEED CHECKING:

Others have addressed possible reasons for lower than expected speed.

CURRENT SPEED:

Noting the rate of change of the free space on the target drive will tell you the current copying rate. While doing this with say 10 seconds between checks, giving it say a minute will give you a better average.

You can do this with whatever Windows tool suits or, by using my preferred option, of using what nowadays passes as a "DOS Window".

       dir target:\

should be adequate.

TRANSFER SPEED TO DATE:

You can find how many files and what data volume have been copied so far with eg

     dir target:\destination_folder /s

This will slow the copying rate during the dir process.
Subsequent DIRs will usually be faster due to caching of the majority of the result.

---

TRANSFER SPEED ADEQUACY:

58 GB/18 hours is somewhat under 1 MB/second, which is low for any USB2 - USB3 copying process. The size of the actual files transferred can be accessed as above - allowing you to tell if the rate is even far slower than 1 MB/s.

Worst case transfer speeds that I have experienced on a modern machine (OK i7) have been in the 501- MB/s

Possible (only) helps:

I am not familiar with Robocopy.
I use a very old (2001) version of XXCOPY for similar purposes as it works well and the licence terms suit my needs. XXCOPY has a flag "/nX0" which disables the short-name-preservation feature in some cases. In some situations this very greatly speeds up transfer rate. Some similar setting in Robocopy may help.

I also assign source and destination paths to dummy drive letters using Ye Olde DOS Subst command

        eg subst a: pathname  

This allows copy of some files which otherwise fail to copy - including those with extremely (stupidly) long file names.

============================

Possible interest.
The question related to the use of Robocopy.
The following script uses XXCOPY to perform a similar task.
Existing files are not recopied.
Y: and Z: are used in place of direct source and destination parameters to eliminate certain copying errors (filename tooooooooo long).
I'm uncertain why this works, but it does.

    subst y: /D ; release mapped drive
    subst y: Src_Drive:\source_path

    subst z: /D ; release mapped drive
    subst z: Dest_Drive:\destination_path

    **xxcopy /bb /s /h /nx0 y:\*.*  z:\**

rem /bb - if not exist
rem /s  - all subdirs
rem /h  - incl hidden files
rem nx0 - don't preserve short file name