3

This question is a bit of an odd one, I'll try to be as detailed as I can be.

I'm repurposing an old computer to serve as an in home backup solution. The final resting place for the data is an external G-Drive connected via USB 2.0 (I told you it's an old computer.) I would like to figure out how to use some space on the computer's new shiny SSD drive to buffer writes to the external drive. I am using OpenSUSE Leap 15, with a server installation.

So far I have found bcache, which I could set to write-back mode and get something similar to what I'm looking for. The problem is the partition I would be buffering uses btrfs (transparent compression) and there have been issues reported using the two together. The arch wiki says that this was fixed in 3.9 but I don't find that anywhere in the linked source.

There's also lvmcache. The only reference to writeback that I see is a brief blip in the man page, without much further explanation. I'm also already using lvm to divvy up the SSD into various partitions with thin pools, so I would be worried that using lvmcache would require me to either create a new dedicated lvm-vg across the two drives, to prevent the thin pools from spreading to the external. Additionally there are notes in the BTRFS wiki that using BTRFS on top of anything block-level could cause problems.

DM-Cache is another highly configurable option that I am still in the process of learning. As it works on the block level, it is still possible that it may conflict with btrfs.

The final option is setting up a SLOG based ZIL with zfs. This source says that with a 1Gbps connection the largest a SLOG would have to be is .625 GB, because it would flush every 5 seconds. However, based on extensive testing, the highest sustained write speed I can get to this drive is 30MB/s. This would imply that it would take 20 seconds to dump a .625GB slog to the ZFS drive. (Update: Based on This source and this source the SLOG device is not intended to boost throughput, just to reduce latency.) (Update to the Update: In the comments Dan clarified what is meant by this. It is still likely that the ZIL could be utilized in this scenario.)

I know that large writes will inevitably be bottlenecked by the external connection. I figure that anything on the order of tens of GB will end up at 30MB/s. My goal is not to solve that problem. I'll likely use rsync to send data to the external, so I would like to speed up most transfers of less than 10GB.

I'm basically curious as to if anyone else has done something similar to this, and has recommendations, or can point to a specific option as being better than others. Right now I'm considering the ZFS/SLOG option to be the best one, using a 10GB slog drive, but I have no idea how that large of a drive will play with the 5 second flush.

tl;dr: What's the best way (though I know not all of this is exactly recommended) to buffer about 10GB worth of data on an internal drive that will then be transferred to an external drive?

|improve this question
  • 1
    You have the weirdest problems, the fanciest solutions and well researched ideas. I love it! – Ricardo S. Nov 13 '18 at 8:15
  • 1
    I've not come across this problem before, but have you looked at dm-cache? (en.m.wikipedia.org/wiki/Dm-cache) – davidgo Nov 13 '18 at 9:26
  • @RicardoS. Hahaha, thanks, I do what I can. – Zyra Nov 14 '18 at 1:49
  • @davidgo LVMCache is based on dm-cache, which would potentially open up the same issues caused by block level solutions + btrfs. That being said, there seems to be a lot of configurability within dm-cache, including throttling the data transference between cache and origin. This means that if I can solve the transparent compression problem, dm-cache might be a better block layer device than the other two. Thanks for the input! :) – Zyra Nov 14 '18 at 1:49
2

I believe any of the solutions you mentioned would work fine (minus the interactions between bcache / LVM and BtrFS -- I don't know more than you do there). The choice probably comes down to what's the easiest for you to understand, set up, and maintain.

Personally, I would choose ZFS because:

  1. I have a lot of experience with it.
  2. It provides an all-in-one solution for this use case (compression plus using SSDs to cache reads and/or writes).
  3. It provides a bunch of other fancy features (snapshots and clones, dedup, RAID-Z, replication with zfs send, encryption, compressed RAM cache, ...) that you may find useful in the future even if you don't use them now. There's also a healthy open source community that's constantly adding new features.
|improve this answer
  • Do you think expanding the SLOG size would help to prevent bottlenecks? The only concern that I have with ZFS is that the ZIL will flush every 5 seconds. I'm unsure if that means that writes are halted during flush, or what happens. I'll have to read more into it. ZFS was what I was leaning toward as well. – Zyra Nov 14 '18 at 1:39
  • 1
    @Zyra: Right -- using a slog (or any write cache) will speed up writes until you've filled it up, at then everything will slow down to the speed of the main pool while things are pushed out of the cache. This does improve throughput when your required throughput spikes above what the main pool can offer, but if your workload consistently needs higher throughput than your main pool, the only way to use a faster main pool. Regarding your other question, writes are not blocked on the buffer flushing, they are only blocked on earlier writes to the ZIL / slog. – Dan Nov 14 '18 at 2:17
  • 1
    Maybe this is being approached from the wrong end. As I understand you're both discussing how to speed up and stabilize the cache by way of picking which fs works best for the intended purpose, while you still have the glaring limitation of the USB 2.0 connection. The mentioned 30MB/s. It might be worth to add a 2nd USB drive and have both extDrives set as RAID 0.. or a Stripe. Wikipedia claims this increases performance. I don't think it's too much of a leap to think that having two drives onto which the cache can offload will increase performance. – Ricardo S. Nov 14 '18 at 7:39
  • 2
    @RicardoS. Definitely — adding additional drives and putting them into a striped zpool, or using an internal hard drive instead of a USB-connected drive, would help performance of the main pool. Write bandwidth scales approximately linearly with the number of drives in ZFS (either for striping or for more complex configurations like RAID-Z that have more redundancy to protect against disk failures), and probably for most other striping solutions as well. – Dan Nov 14 '18 at 7:44
  • 1
    @Zyra Right, striping is configured via the configuration you give to zpool create. You can also add more disks to a pool via zpool add today (example) but you won't be able to add disks to a RAID-Z until after the project you mentioned is released. – Dan Nov 15 '18 at 1:24

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.