3

When using an SSD (flash) cache on a HDD (hard disk) filesystem (using bcache or lvmcache on dm-cache), it can happen that some of the blocks necessary for some global operation (maybe a find) are aged out of the cache making access very slow. This is because typical filesystem metadata access patterns are small random reads where HDDs are particularly slow (more than 100 times slower than SSDs).

Forcing all metadata (inodes, bitmaps, dentries, journal and possibly small files) on SSD by design may solve the problem of slow and unpredictable access times, while keeping files (possibly those larger than some threshold) on HDD makes the solution workable even for large filesystems.
(If both SSD and HDD are mirrored, there is no elevated risk of data loss.)

Are there any Linux filesystems able to use a small non-rotational device for metadata and a large rotational device for file data ?

While btrfs is able to force one mirror of metadata to a SSD, it seems unable to force file contents out of the SSD. And I'm unsure whether it is able to read metadata only from the SSD. (Metadata is mirrored and thus also available from the HDD, but that mirrored metadata shall only be used in case of SSD failures.)

Improve this question

2 Answers 2

Reset to default
2

In ZFS you can use Special Devices to store metadata, deduplication tables, and optionally small files up to a user-defined size.

Special Allocation Class

The allocations in the special class are dedicated to specific block types. By default this includes all metadata, the indirect blocks of user data, and any deduplication tables. The class can also be provisioned to accept small file blocks.

A pool must always have at least one normal (non-dedup/special) vdev before other devices can be assigned to the special class. If the special class becomes full, then allocations intended for it will spill back into the normal class.

https://openzfs.github.io/openzfs-docs/man/8/zpoolconcepts.8.html#Special_Allocation_Class

So this can be used to accelerate disk accesses using an SSD disk. You can choose to mirror multiple metadata devices for redundancy, or just use the SSD as the only metadata storage. AFAIK when that special device is full then metadata will be stored in the main storage so things will work just as normal

Special VDEV Class

In ZFS 0.8 and later, it is possible to configure a Special VDEV class to preferentially store filesystem metadata, and optionally the Data Deduplication Table (DDT), and small filesystem blocks. This allows, for example, to create a Special VDEV on fast solid-state storage to store the metadata, while the regular file data is stored on spinning disks. This speeds up metadata-intensive operations such as filesystem traversal, scrub, and resilver, without the expense of storing the entire filesystem on solid-state storage.

https://en.wikipedia.org/wiki/ZFS#Special_VDEV_Class

ZFS Special Device

A special device can improve the speed of a pool consisting of slow spinning hard disks with a lot of metadata changes. For example workloads that involve creating, updating or deleting a large number of files will benefit from the presence of a special device. ZFS datasets can also be configured to store whole small files on the special device which can further improve the performance. Use fast SSDs for the special device.

https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_special_device

Example on how to set up this

Adding the special vdev to an existing zpool

#implied stripe of mirrors, for this special device 
zpool add elbereth -o ashift=12 special mirror /dev/nvme0n1 /dev/nvme1n1 mirror /dev/nvme2n1 /dev/nvme3n1

Setting the Small Blocks size:

# zfs set special_small_blocks=128K elbereth

Or if you have a “videos” dataset like ours:

# zfs set special_small_blocks=128K elbereth/videos

ZFS Metadata Special Device: Z

See also

Improve this answer
0

Bcache is

... a Linux kernel block layer cache. It allows one or more fast disk drives such as flash-based solid state drives (SSDs) to act as a cache for one or more slower hard disk drives.

And its descendant Bcachefs, a modern filesystem comparable to Btrfs and ZFS, also supports multiple target device types: 3 storage + 1 metadata targets

In bcachefs these activities are categorised as target options. Four target options exist which may be set at the filesystem level (at format time, at mount time, or at runtime via sysfs), or on a particular file or directory:

  • foreground target: normal foreground data writes, and metadata if metadata target is not set
  • metadata target: btree writes
  • background target: If set, user data (not metadata) will be moved to this target in the background
  • promote target: If set, a cached copy will be added to this target on read, if none exists

Labels and target options

You can specify a special metadata device like these examples

# bcachefs format /dev/sd[ab] --foreground_target /dev/sda --promote_target /dev/sda \
    --background_target /dev/sdb --metadata_target /dev/sda

# bcachefs format --compression=zstd \
    --encrypted \
    --replicas=2 \
    --label=hdd.hdd1 /dev/sdc \
    --label=hdd.hdd2 /dev/sdd \
    --label=hdd.hdd3 /dev/sde \
    --label=ssd.ssd1 /dev/sdl \
    --label=ssd.ssd2 /dev/sdm \
    --foreground_target=ssd \
    --promote_target=ssd \
    --background_target=hdd \
    --metadata_target=ssd

However using an SSD just for metadata is usually not very efficient, because metadata constitute a very small part of the whole data set, so a 512GB SSD is only useful for huge block devices like 25600GB or more

See also

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .