3

I'm considering moving my personal home data to ZFS on a Debian PC to take advantage of ZFS features such as data integrity and checksums, deduplication, and snapshots. The box has to be more than just a NAS, so I'm not interested in dedicating the box to something like FreeNAS.

In doing so, I'm also thinking about how I'm going to back up the data nightly and periodically take it off-site. My current setup is described later in this post. I would appreciate any pros and cons about the third approach described below for backups.

(1) Since I'm new to and unfamiliar with ZFS, I'm considering playing it safe by putting my live data in ZFS pools but continuing to use Dirvish to backup that data onto a non-ZFS backup drive.

(2) I've also learned about setting up cron jobs for periodic snapshots that could be as frequent as hourly snapshots, so that got me thinking about setting up the live data drive taking hourly or daily snapshots, and then sending/receiving to the backup drive that's also formatted for ZFS.

(3) Finally, it recently hit me that I might not need to do anything this complex with ZFS. What if I set up my two drives in a mirrored vdev configuration similar to RAID1, and then set up hourly snapshots? This way, it seems I would have an instantaneous backup and I'm protected against the dreaded RAID-is-not-a-backup:

/bin/rm -Rf /*

With #3, if either of the drives goes down, I have the mirror available with which to continue. As I mention below, I'm I can live without the high availability of RAID, but in this case it looks like I might be getting it for free.

The only concern I would have would be my off-line, off-site drive described below. Would it be a problem to periodically rotate one mirrored drive with a spare that's several weeks or months old?

Regarding my current setup, I currently keep everything on an Ubuntu box in probably an ordinary ext3 partition. I don't use RAID anymore; I don't have any personal need for high availability; if things go down, I can afford the time to restore from backup.

I use dirvish to backup my data, laptop, and anything else of value nightly with snapshots going back about two months.

Periodically, I rotate my backup drives so I can take it off-site and off-line. If I lose my primary drive and my live backup drive, at least I'll still have the off-site drive even if it's several weeks old.

I'm not considering maintaining an array of drives; I'd prefer to have a single drive for all data, and another single drive for the entire backup. This makes it easy for me to periodically rotate a single off-site backup drive with the backup drive currently plugged-in.

Thanks.

EDIT: I gave jlliagre's #2 a try on a VM running Debian with ZFS on Linux installed using commands described here:

http://www.bsdnow.tv/tutorials/zfs

I started by creating three files to simulate my three disks:

# cd /mnt/sda3
# truncate -s 2G primary
# truncate -s 2G rotating_mirror1
# truncate -s 2G rotating_mirror2

As stated, I created a mirrored pool using only one of the mirror files:

# zpool create mypool mirror /mnt/sda3/primary /mnt/sda3/rotating_mirror1
# zfs list
NAME     USED  AVAIL  REFER  MOUNTPOINT
mypool   106K  1.95G    30K  /mypool
# zpool status -v mypool
  pool: mypool
 state: ONLINE
  scan: none requested
config:

    NAME                            STATE     READ WRITE CKSUM
    mypool                          ONLINE       0     0     0
      mirror-0                      ONLINE       0     0     0
        /mnt/sda3/primary           ONLINE       0     0     0
        /mnt/sda3/rotating_mirror1  ONLINE       0     0     0

errors: No known data errors

Next, I tried setting up automated snapshots using the script referenced here:

http://alblue.bandlem.com/2008/11/crontab-generated-zfs-snapshots.html

I created a crontab file for root:

# crontab -e

I pasted and modified to the following from the link above. I'm no cron expert, so I wonder if there's any race condition in this:

@reboot /sbin/zpool scrub mypool
@daily  /sbin/zpool scrub mypool
@hourly /sbin/zpool status mypool | /bin/egrep -q "scrub completed|none requested" && /sbin/zfs snapshot -r mypool@AutoH-`date +"\%FT\%H:\%M"`
@daily  /sbin/zfs snapshot -r mypool@AutoD-`date +"\%F"`
@weekly /sbin/zfs snapshot -r mypool@AutoW-`date +"\%Y-\%U"`
@monthly /sbin/zfs snapshot -r mypool@AutoM-`date +"\%Y-\%m"`
@yearly /sbin/zfs snapshot -r mypool@AutoY-`date +"\%Y"`
# do a spot of housecleaning - somewhat assumes the daily ones have run ..
@hourly /sbin/zpool status mypool | /bin/egrep -q "scrub completed|none requested" && /sbin/zfs list -t snapshot -o name | /bin/grep mypool@AutoH- | /usr/bin/sort -r | /usr/bin/tail -n +26 | /usr/bin/xargs -n 1 /sbin/zfs destroy -r
@daily  /sbin/zfs list -t snapshot -o name | /bin/grep mypool@AutoD- | /usr/bin/sort -r | /usr/bin/tail -n +9 | /usr/bin/xargs -n 1 /sbin/zfs destroy -r
@weekly /sbin/zfs list -t snapshot -o name | /bin/grep mypool@AutoW- | /usr/bin/sort -r | /usr/bin/tail -n +7 | /usr/bin/xargs -n 1 /sbin/zfs destroy -r
@monthly /sbin/zfs list -t snapshot -o name | /bin/grep mypool@AutoM- | /usr/bin/sort -r | /usr/bin/tail -n +14 | /usr/bin/xargs -n 1 /sbin/zfs destroy -r

I don't think this will get me my two-months of backups, but it's a start.

At this point, I started playing with the example in the first link to create a file, create a snapshot, and then modify the file.

Continuing with jlliagre's recommendation, I wanted to swap out rotating_mirror1 and replace with rotating_mirror2. (In the real box, I would expect to power down the box to remove a SATA mirror1 drive to replace with the mirror2 drive in its place.)

# zpool split mypool mypool_bak /mnt/sda3/rotating_mirror1
# zpool status
  pool: mypool
 state: ONLINE
  scan: none requested
config:

    NAME                 STATE     READ WRITE CKSUM
    mypool               ONLINE       0     0     0
      /mnt/sda3/primary  ONLINE       0     0     0

errors: No known data errors

At this point, I got into some trouble; I couldn't import mypool_bak nor do anything else with rotating_mirror1. I ended up trying to create mypool_bak, saw nothing was in there, destroying it, and then attaching rotating_mirror1 back to primary.

# zpool create -f mypool_bak /mnt/sda3/rotating_mirror1
# zpool destroy mypool_bak
# zpool attach mypool /mnt/sda3/primary /mnt/sda3/rotating_mirror1 

I'm purely assuming that rotating_mirror1 at this point synchronized with any changes to primary.

I tried to split again:

# zpool split mypool mypool2 /mnt/sda3/rotating_mirror1

Again the same problem, so I reattached:

# zpool attach -f mypool /mnt/sda3/primary /mnt/sda3/rotating_mirror1

Tried again with the -R option. This looks like it worked better:

# zpool split -R /mypool2a mypool mypool2 /mnt/sda3/rotating_mirror1
# ls /mypool2a/mypool2/
file1.txt  file2.txt  file3.txt  somefile
# zpool status
  pool: mypool
 state: ONLINE
  scan: resilvered 566K in 0h0m with 0 errors on Wed Dec  3 00:18:28 2014
config:

    NAME                 STATE     READ WRITE CKSUM
    mypool               ONLINE       0     0     0
      /mnt/sda3/primary  ONLINE       0     0     0

errors: No known data errors

  pool: mypool2
 state: ONLINE
  scan: resilvered 566K in 0h0m with 0 errors on Wed Dec  3 00:18:28 2014
config:

    NAME                          STATE     READ WRITE CKSUM
    mypool2                       ONLINE       0     0     0
      /mnt/sda3/rotating_mirror1  ONLINE       0     0     0

errors: No known data errors

Next:

# zpool export mypool2
# rmdir /mypool2a

It looks like I should now be able to bring in rotating_mirror2:

# zpool attach mypool /mnt/sda3/primary /mnt/sda3/rotating_mirror2
# zpool status
  pool: mypool
 state: ONLINE
  scan: resilvered 524K in 0h0m with 0 errors on Wed Dec  3 00:25:36 2014
config:

    NAME                            STATE     READ WRITE CKSUM
    mypool                          ONLINE       0     0     0
      mirror-0                      ONLINE       0     0     0
        /mnt/sda3/primary           ONLINE       0     0     0
        /mnt/sda3/rotating_mirror2  ONLINE       0     0     0

errors: No known data errors

At this point, I created a couple more files and snapshots to simulate the disks being used some more over time.

Now I wanted to remove rotating_mirror2, but I didn't want to bring rotating_mirror1 back in just yet.

# zpool split -R /mypool2a mypool mypool2 /mnt/sda3/rotating_mirror2
# zpool export mypool2
# rmdir /mypool2a

I would imagine if I try to bring rotating_mirror1 back in at this point, all its contents would be destroyed and the current primary would be mirrored onto rotating_mirror1.

What if instead primary went down and needed to be restored from rotating_mirror1 at this point? How can I use rotating_mirror1 to rebuild primary?

# zpool destroy mypool
# zpool import -d /mnt/sda3/ -N mypool2 mypool
# zpool status
  pool: mypool
 state: ONLINE
  scan: resilvered 524K in 0h0m with 0 errors on Wed Dec  3 00:25:36 2014
config:

    NAME                          STATE     READ WRITE CKSUM
    mypool                        ONLINE       0     0     0
      /mnt/sda3/rotating_mirror1  ONLINE       0     0     0

errors: No known data errors
# zfs set mountpoint=/mypool mypool
# ls /mypool

Funny. At this point, I would've expected to see the files I had on the first mirror, but instead I see nothing.

EDIT: I restarted the steps all over again and made a couple minor tweaks now that I know a bit more. I didn't do all the steps below, but they're all here for completeness. I think I got it this time.

# cd /mnt/sda3
# truncate -s 2G primary
# truncate -s 2G rotating_mirror1
# truncate -s 2G rotating_mirror2
# zpool create mypool mirror /mnt/sda3/primary /mnt/sda3/rotating_mirror1
# crontab -e
# echo (Paste crontab snippet from above.)

Use the pool.

# echo QUESTION Should a snapshot be created here?
# zpool split -R /mypool2 mypool mypool2 /mnt/sda3/rotating_mirror1
# zpool export mypool2
# rmdir /mypool2
# zpool attach mypool /mnt/sda3/primary /mnt/sda3/rotating_mirror2

Use the pool some more.

# echo QUESTION Should a snapshot be created here?
# zpool split -R /mypool2 mypool mypool2 /mnt/sda3/rotating_mirror2
# zpool export mypool2
# rmdir /mypool2

Do something bad enough to mypool that it needs to be recreated.

# echo QUESTION Should a backup of rotating_mirror2 be made prior to restoration in case something bad happens, or is there any way to bring in its contents read-only?
# zpool export mypool
# zpool import -d /mnt/sda3/ mypool2 mypool
# zpool attach mypool /mnt/sda3/rotating_mirror2 /mnt/sda3/primary

Note that once the import command has run, both primary and rotating_mirror2 have the same name for the pool, so any subsequent import commands in this example won't work as listed above. Instead, the pool will need to be imported by numeric ID:

# zpool import -d /mnt/sda3/
   pool: mypool
     id: 17080681945292377797
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

    mypool                        ONLINE
      /mnt/sda3/rotating_mirror2  ONLINE

   pool: mypool
     id: 4810117573923218177
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

    mypool               ONLINE
      /mnt/sda3/primary  ONLINE

# zpool import -d /mnt/sda3/ 4810117573923218177
# zpool export mypool
# zpool import -d /mnt/sda3/ 17080681945292377797

Honestly, though, this gives me a bit of a dirty feeling because importing the mirror is causing it to be modified since its pool name has changed and because it's now live and able to be modified while primary is being reconstructed.

I can understand that being fine for a mirror, but recall I want to treat these mirrors as live backups when they're "plugged-in".

I haven't tried this, but if there are no objections, I think I would prefer not to import the rotating_mirror2 and instead do something like:

# dd if=/mnt/sda3/rotating_mirror2 of=/mnt/sda3/primary bs=...

When that's done, I would hope I can import primary directly and everyone would be happy.

One final note: I learned that I can have more than one mirror, so if I really wanted, I could set up four drives instead of three in which two would be always-live "primary" mirrors of each other, and rotating_mirror1 and rotating_mirror2 would be a third drive instead of the second drive.

Thanks.

  • This site isn't really designed for this type of broad question. It is designed for specific questions that have specific, fact-based answers. Opinion-based answers are specifically off-topic, as are questions that would require volumes to answer properly. Can you break this question into specific, bite-sized things you need to know? – fixer1234 Dec 6 '14 at 9:17
  • So is the problem solved or are you still looking for an answer? You began the question, " I'm considering moving my personal home data to ZFS...". Is all of that data included in this question? :-) – fixer1234 Dec 7 '14 at 6:49
  • Yes, unless there are any objections to the write-up. jlliagre answered my original question, but it was too high-level for a ZFS newbie like me. I think I've been able to prototype what I want to do in my VM; the next step is to try it out with physical hardware over the next couple weeks when I get a chance. I'll migrate the real data once I'm comfortable with all this. Thanks to all for the help and support! – jia103 Dec 7 '14 at 7:38
  • @jia103 If you find my answer too high level, feel free to comment it to ask for more details or explanations ! – jlliagre Dec 7 '14 at 8:58
  • It's alright. I appreciate the direction; I was able to perform the steps above to figure out how to do this based on your thoughts. Thanks. – jia103 Dec 7 '14 at 16:03
1

1: If you don't want ZFS redundancy, you can:

  • Use your first disk as your active pool.
  • Schedule snapshots to allow fast rollbacks in case of human errors like rm -rf ones.
  • Create a backup pool on your second disk
  • Regularly update this backup pool from first pool incremental snapshots: zfs send ... | zfs receive ....

The backup pool need to be created with an alternate root to avoid mount point clashes.

You can swap the backup disk with another one to store it elsewhere. Just export the backup pool before removing the disk, create an alternate backup pool on the other disk and do an initial full backup of the first pool.

2: If you want redundancy, the way to do it would be to used a mirrored pool and, when you want to swap disks, split the pool with the zpool split command, remove to newly created pool disk, insert the alternate disk, destroy the alternate pool and configure mirroring with the added disk.

#2 is safer as there is data redundancy so would be the one I recommend.

  • Actually I do have one question. Periodic snapshots will be made anyway, but would it be a good practice to create a snapshot just before splitting? My understanding is that snapshots are cheap anyway, so it shouldn't hurt to create a snapshot just in case, right? Thanks. – jia103 Dec 7 '14 at 16:14
  • Indeed, creating a snapshot is a lightweight operation with ZFS. Glad to see you progressed a lot with your experiments. I would probably recommend for you to avoid going the dd way you are considering at the end of your question. That would be slow and might confuse ZFS more than helping to avoid your import issue. Finally, a three side mirror is indeed a good way to make sure your data is always redundant. – jlliagre Dec 7 '14 at 22:54
  • "The backup pool need to be created with an alternate root to avoid mount point clashes." If the backup pool doesn't need to have its file systems usable during the zfs receive process, then zpool import -N tank is a viable alternative. (-N means only import the pool, don't mount any file systems.) – a CVn Jun 14 '15 at 13:01
  • @MichaelKjörling You are correct indeed although using an alternate root will eventually be required should the backup files need to be accessed on the receiver side. – jlliagre Jun 14 '15 at 13:47
  • @jlliagre True; the applicability of -N depends very much on the specific needs at the time. – a CVn Jun 14 '15 at 17:23
0

(3) Finally, it recently hit me that I might not need to do anything this complex with ZFS. What if I set up my two drives in a mirrored vdev configuration similar to RAID1, and then set up hourly snapshots? This way, it seems I would have an instantaneous backup and I'm protected against the dreaded RAID-is-not-a-backup: /bin/rm -Rf /*

How does that help you against a command on ZFS level which completely removes all your snapshots, datasets and pools by accident? Obviously RAID-still-is-not-a-backup and ZFS send/receive provide you good tools to not even think about relying on RAID only as a backup.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.