1

I'm trying to replace a 640GB USB2 drive which I use as my Time Machine backup device with a 1TB FireWire 400 drive without losing the current backup history. Given that the 640GB drive is nearly full and the 400Mbps transfer limit of this combination, I realized that this process could take quite some time and might get interrupted in the middle. As a result, I decided to try doing it with rsync instead of Finder (as Apple suggests). After some false starts and some searching online I settled on the following rsync command:

rsync -aHXSvPh --hfs-compression --protect-decmpfs /Volumes/Macintosh\ BK/Backups.backupdb /Volumes/Untitled

However, this command is still causing significant bloat on the destination drive (to the point where I don't expect the contents of the old drive to fit on the new one, despite the new drive being about 1.5 times larger). Are there any rsync options which I'm missing which would eliminate this bloat (I'm using v3.1.2 protocol version 31)?

It has also occurred to me that perhaps I'm using the wrong tool for the job. Would a block copy tool like dd be more appropriate? If so, how would I set that up so as to make the process resumable in the event of an interruption (such as one caused by a full system crash, something which happened to me twice while running the rsync command)? I've never used dd before and so am unfamiliar with its abilities (but do have access to both the version that comes packaged with Mac OSX and the GNU version 8.25).

  • I answered below, but then realized your rsync invocation should have preserved hardlinks given the -H. Is your destination volume HFS+? If you run rsync --version does it list hardlinks in the capabilities statement? What happens if you use the system rsync instead of the one you're using? – Spiff Mar 21 '16 at 17:42
  • Another thought: -a implies no -H. I wonder if the presence of -a is overriding your explicit -H. – Spiff Mar 21 '16 at 17:47
  • I've done this before for similar size backup images. Just use the Finder, and keep your Mac and drives from going to sleep, and you should be fine. – jimtut Mar 22 '16 at 16:30
2

dd doesn't adjust the various volume data structures based on the target volume size, so it's only appropriate if the source and destination volumes are exactly the same size. Generally, I recommend asr instead, but it doesn't have a good way to continue after a crash.

So, one possibility would be to shrink the target volume to match the source, use dd to copy the raw volume, then expand the target back to the full 1TB. I haven't tested this, but I think this is the process you'd need:

  1. diskutil list will list the volume's device identifiers (e.g. disk2s3 is the third slice (partition) of physical disk #2)
  2. diskutil info <sourcevolumeid> will list the source volume size in bytes (along with lots of other info)
  3. diskutil resizeVolume <targetvolumeid> <sourcevolumesize>B (the "B" means "bytes" -- see the "SIZES" section of the diskutil man page).
  4. diskutil unmount <sourcevolumeid> and diskutil unmount <targetvolumeid> -- don't use dd on mounted volumes!
  5. sudo dd if=/dev/r<sourcevolumeid> of=/dev/r<targetvolumeid> to do the copy. Note the "r" prefix on the device names -- this bypasses the OS disk buffers, and in my experience makes dd run much faster. Be very careful to get the volume IDs right, or you may copy a blank volume over your backup!
  6. Finally, use either diskutil resizeVolume or Disk Utility to expand the target volume out to the full disk's size.

Oh, and a warning: this process assumes neither the source or destination is being managed by Core Storage. If they are (e.g. if they're encrypted), things get a bit more complicated.

| improve this answer | |
  • What I've read about dd indicates that getting the bs argument right can greatly speed up the process. Obviously, though, I don't want to be testing to determine the right size with the whole volume. If I were to use something like the scripts here or here to determine the optimal value for bs, where would I insert that step and would it be necessary to ensure the disks are unmounted during that process? – rpspringuel Mar 21 '16 at 20:30
  • I'd do that after dismounting the volumes (i.e. between steps 4 and 5). You can also test with the buffered device (without the "r" prefix), but in my experience that's always significantly slower. Be careful you don't test writing to the source volume! – Gordon Davisson Mar 21 '16 at 20:43
  • This process worked with one modification: After performing step 5, the ownership of the new volume had changed to match the ownership of the Time Machine source volume. Thus in order to resize the volume I first had to change it's ownership using chown. Once I owned the volume, I was allowed to resize it and could finish the process. – rpspringuel Mar 26 '16 at 20:12
  • Is this really true? i swear i've used dd to dump smaller backups onto bigger disks in the past. But maybe my memory is failing me. – deweydb Jul 14 '17 at 5:05
  • @deweydb dding to a larger partition will sort of work. See here for an example of the trouble it can cause. – Gordon Davisson Jul 14 '17 at 5:33
2

The short answer is that Time Machine makes heavy use of hardlinks, which is where one underlying file – one stream of bytes – appears as multiple filenames in multiple directories on the filesystem. If you do a copy that isn't savvy about hardlinks, then that underlying file will get copied multiple times (once for each time it's hardlinked).

So let's say you have a 1GB disk image file, that's been there from the beginning, so it shows up as a hardlink in all of your periodic Time Machine backups, and you have 100 Time Machine backups. If you copy that without being smart about hardlinks, You'll end up with 100 copies of that 1GB file, for 100GB total, instead of just 1GB with 100 hardlinks pointing at it.

I'm sure Apple knows their Finder-copy advice preserved hardlinks properly. I would expect that Disk Utility's "Restore" functionality does as well. Don't let the "Restore" name fool you, it's not just for restoring from backups; it's Disk Utility's way to do a block copy from one volume (or disk image) to another.

Updated to add:
Hmm, rsync -H should have preserved hardlinks. Are you sure your destination volume is HFS+, or another filesystem that OS X trusts with hardlinks?

| improve this answer | |
  • 1
    rsync -H will preserve hardlinked files, but Time Machine also uses hard-linked directories -- that's something most unixes don't support, so I'm pretty sure rsync doesn't even contemplate them as possibilities, let alone handle them properly. – Gordon Davisson Mar 21 '16 at 19:08
  • @GordonDavisson That's probably a better answer than mine. Why don't you post that separately and I'll delete mine? – Spiff Mar 21 '16 at 19:16
  • I'd leave yours -- I'm pretty sure it's correct about the core problem. Also, I just posted an alternate suggestion. – Gordon Davisson Mar 21 '16 at 19:34
  • One of my false starts included leaving off the -H, so I know that rsync is handling the hardlinked files correctly. Of course, that doesn't mean anything if the issues is with hardlinked directories as @GordonDavisson suggests. – rpspringuel Mar 21 '16 at 20:05
  • Also, I am sure that the volume is HFS+. Switching the drive filesystem to that was the first thing I did when I got it. – rpspringuel Mar 21 '16 at 20:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.