I Have a "home lab" system (Ubuntu 22.04 on i3-9300 with 32G ecc-ram, 10G of witch is dedicated to hugepages) currently holding 2 zfs raidz1 pools, one with 3x2TB WD-RED and nvme as log and cache device, and the second one with 3x1TB HDD (planing to extend this one to 3x12TB) currently used as backup only

bartoszek@home-lab:~$ zpool list -v
NAME                                           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
storage-pool                                  2.72T   585G  2.15T        -         -     2%    21%  1.00x    ONLINE  -
  raidz1-0                                    2.72T   585G  2.15T        -         -     2%  21.0%      -    ONLINE
    sdb                                           -      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J7LNUNJS      -      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD10EZEX-00RKKA0_WD-WMC1S5929034      -      -      -        -         -      -      -      -    ONLINE
system-pool                                   5.45T  3.46T  1.99T        -         -     3%    63%  1.00x    ONLINE  -
  raidz1-0                                    5.45T  3.46T  1.99T        -         -     3%  63.5%      -    ONLINE
    ata-WDC_WD20EFRX-68EUZN0_WD-WMC4M1371094      -      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M6HSDJXY      -      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M1628970      -      -      -        -         -      -      -      -    ONLINE
logs                                              -      -      -        -         -      -      -      -  -
  nvme0n1p2                                   15.5G  1.42M  15.5G        -         -     0%  0.00%      -    ONLINE
cache                                             -      -      -        -         -      -      -      -  -
  nvme0n1p3                                   64.0G  54.6G  9.44G        -         -     0%  85.2%      -    ONLINE

The whole system is used mainly for Nextcloud, Plex, zoneminder and some Virtualization (GPU passthrough to a Windows10 vm, some other stuff) so nothing really special (please notice no torrent)

Because some of my use cases require load of ram to work (VM/Zoneminder) I have limited resources for ZFS caches (and as for now no plan to extend RAM) so I want to squeeze as much performance as possible from the HW I currently own. The workload I have can be split into basically two main categories, write a lot of data/read almost none of it or read once (zoneminder/plex/backups) and write once/read multiple times (games on VM/nextcloud) I decided to disable cache for first group and enable for rest. So to recap

  • Nextcloud -> primarycache=all secondarycache=all
  • VM -> primarycache=all secondarycache=all
  • rest of stuff-> primarycache=none secondarycache=metadata

But then I realize, that the second category can be divided further to 'I need it as fast as possible' (Nextcloud) and 'just don't be annoyingly slow' (Ocasional gaming on my vm). And just to be clear, nextcloud is NOT used more often I just want to prioritize ARC to hold as much Nextcloud data in ram as possible and in the same time cache game data on NVME, because more annoying for me is slow nextcluod, then a little bit longer loading times of some games. And started to make some tests by creating a temporary dataset

sudo zfs create -o recordsize=1M -o primarycache=none -o secondarycache=all -o mountpoint=/srv/tmp_1M_noprimarycache system-pool/ENC/tmp_1M_noprimarycache

and then to put some read only stress on it I tried

sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=1024k --iodepth=2 --size=4G --readwrite=randread --loops=10

And noticed that my cache device is NEWER used (zpool iostat shows no IO on NVME drive) I assume that this is because L2ARC is feed by "drops" from main ARC but why is this configuration even valid? And how to configure ZFS to use L2ARC without main ARC?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.