18

How can make a full copy of the contents of a btrfs filesystem? By full copy I mean not only the current data, but also different subvolumes with their snapshots, ideally preserving their CoW structures (i.e.: not duplicating blocks with the same content.

It seems a block-level copy (such as with dd) is not a good idea, since it duplicates the UUID, and there isn't a way to easily change it, apparently.

6 Answers 6

4

Option 1 - Dumb data copy then change UUID

Ensure that source partition is unmounted and will not be automounted.

Use either dd (slow, dumb) or partclone.btrfs -b -s /dev/src -o /dev/target

Use btrfstune -u to change UUID after copy and before mounting.

Data loss warning: Do NOT try to (auto)mount either original or copy until the UUID has changed


Option 2 - btrfs-clone

I have not personally tried btrfs-clone, but it purports to clone an existing BTRFS file system to a new one, cloning each subvolume in order.

1
19

I have not found any ready-made solution as of today (2016-05-06), but solved the problem for my purposes, including Copy-on-Write handling. The steps to "clone" /source to /target are:

  1. Get a list of subvolumes ordered by ogen: btrfs subvolume list -qu --sort ogen /source. Sorting is probably enough to guarantee that snapshots or subvolumes which depend on previous ones are handled first. This is important for dealing with Copy-on-Write, because we need to have the base volumes transferred first.

  2. Make all subvolumes read-only using btrfs property set -ts /source/some-volume ro true.

  3. Now, for each subvolume from the list above, starting at the top, do the following:

    1. If the volume does not have a parent UUID (displayed as -) or the parent UUID does not exist anymore in the list, run: btrfs send /source/some/volume | btrfs receive /target/some/

    2. If the volume does have a parent UUID which still exists, we should have transferred it already because of --sort ogen and we can use that as a base to avoid data duplication. Hence, find the parent UUID's path in the list and run: btrfs send -p /source/parent/volume/ -c /source/parent/volume/ /source/some/volume/ | btrfs receive /target/some/ (btrfs would probably guess the -p argument automatically, but I prefer to be explicit).

    3. After running one of the above commands make the target and source read-write again: btrfs property set -ts /source/some/volume ro false; btrfs property set -ts /target/some/volume ro false. This step can be skipped if the source has been previously read-only.

This should handle many cases. Caveats:

  1. There might be some complications with respect to ordering when nesting subvolumes/snapshots.

  2. The whole process is obviously more fun when scripted.

  3. btrfs send accepts multiple clone source (-c) arguments. It may be advantageous to not only specify the parent's volume path, but also those of any ancestors or simply any previously sent volumes. It did not make any difference here, but it might — just a guess — help to avoid data duplication in some cases.

  4. I am unsure if any meta information on snapshots or subvolumes is lost along the way, but just about everything interesting else for most use cases should be preserved.

The whole process helped me transfer an 800 GB filesystem with 3.8 GB used (according to df) to a 10 GB image with 3.8 GB used. Transferring without -p and -c would have used about 190 GB, so data duplication was indeed avoided.

3
  • Well written answer, thanks. Can you explain what ogen means?
    – drumfire
    Jun 3, 2016 at 16:09
  • @drumfire ogen is the subvolume's "origin generation". I have to admit that I do not fully understand the differences or whether using the (non-origin) generation would be correct, but assume some test indicated that this worked better (avoided duplication). The generation does seem to get updated when creating snapshots based on a subvolume, ogen doesn't. I would be interested in hearing about some findings. It's probably best to check on IRC or the Btrfs mailing list. Jun 4, 2016 at 14:10
  • 2
    I just took the algorithm of @ThomasLuzat , added some fluff around it (error checking etc) and put it here: github.com/jernst/btrfs-copy-filesystem/blob/master/… . It worked for my problem getting off a corrupted disk, and there are no guarantees it will work for anybody else. But I'm posting this here anyway in case anybody wants to start from somewhere other than scratch to code this. Currently depends on a new UBOS methods but should be easy to port. Oct 1, 2017 at 1:31
11

I have created a python tool which can do this. I did this because I tried @Thomas Luzat's approach in both my own and @Johannes Ernst's implementation, and the used space doubled from 20GB to 40GB in the cloning procedure. I thought something more efficient was needed.

Consider this common file system history:

current ---------------------------------\
             |       |        |          |
           snap4   snap3    snap2      snap1

With Thomas' algorithm, "current" would be cloned first, and all snapshots (being snapshots of former states of "current") would use "current" as clone source / parent. Obviously, it would be better to base snap3 on snap4, snap2 on snap3, etc.

And this is just the tip of the iceberg; finding the "best" clone sources (in terms of space savings) in a btrfs file system with a complex history is a non-trivial problem. I've come up with 3 other strategies to solve this problem, which seem to use space much more efficiently. One has actually resulted in clones size slightly below that of the source.

You can read the details on the github page if you're interested.

2

There is a similar question on unix.stackexchange.com that points to partclone.btrfs, but I do not know any specifics about this.

There is also a discussion on the kernel mailing list, not really looking promising...

2

With btrfs-send, which last I saw, was still experimental patches floating around on the btrfs mailing list.

1
0

Unfortunately btrfs-clone does a weird thing with the root subvolume (all other subvolumes handled fine). It create a snapshot of root subvol first (btrfs send is unable to deal with root subvolume), copies that one to the destination (where it becomes a snapshot too, not root!), and next it moves files in the destination filesystem from the newly created snapshot to the empty root using mv. This is why whole point of it is lost: this mv step copies each file in turn between two subvolumes in the target devices, effectively voiding all the reflinks/compression/whatever, and doing second copy. This is why it is very inefficient (doing double copy) and increases the used space. And this is why it is basically useless when you want to clone a root subvolume, - it definitely is not a clone.

New contributor
Michael Tokarev is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
2
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Nov 26 at 9:55
  • Welcome to Super User! Before answering an old question having an accepted answer (look for green ✓) as well as other answers ensure your answer adds something new or is otherwise helpful in relation to them. Here is a guide on How to Answer. There is also a site tour and a help center. Nov 26 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.