Linux, SSD, BTRFS and Trim

Question

I'm running a Linux system (based on Gentoo) with a BTRFS file system installed on an SSD (Toshiba Q300 with 480GB net). My /etc/fstab looks like:

UUID=14cb9b65-... swap           swap    defaults,noatime, 0       0                                                                                                                                                                                                                           
UUID=cd7d93b3-... /              btrfs   defaults,cache,compress=lzo,subvol=@ 0       1                                                                                                                                                                                          
UUID=cd7d93b3-... /home          btrfs   defaults,noatime,space_cache,compress=lzo,subvol=@home 0       2                                                                                                                                                                                      
UUID=cd7d93b3-... /Data          btrfs   defaults,noatime,space_cache,compress=lzo,subvol=@Data 0       2                                                                                                                                                                                     
UUID=cd7d93b3-... /mnt/rootfs    btrfs   defaults,noatime,space_cache,compress=lzo 0       0                                                                                                                                                                                                   
tmpfs             /tmp           tmpfs   defaults,noatime,mode=1777 0       0                                                                                                                                                                                                                   tmpfs             /proc          proc    defaults                   0       0                                                                                                                                                                                                              
tmpfs             /var/log       tmpfs   defaults,noatime,rw,mode=1777 0    0
tmpfs             /var/tmp       tmpfs   defaults,noatime,rw,mode=1777 0    0                                                                                                                                                                                                              
tmpfs             /var/run        tmpfs   defaults,noatime             0    0 
tmpfs             /var/spool      tmpfs   defaults,noatime             0    0 
tmpfs             /var/lock       tmpfs   defaults,noatime             0    0 
tmpfs             /var/cache      tmpfs   defaults,noatime             0    0
tmpfs             /run            tmpfs   defaults,noatime             0    0
sysfs             /sys            sysfs   defaults                     0    0
tmpfs             /dev/shm        tmpfs   defaults                     0    0
devpts            /dev/pts        devpts  gid=5,mode=620               0    0
devtmpfs          /dev            devtmpfs gid=5,mode=620              0    0

Before, I had an Intel SSD with 240GB net with XFS file system. When I executed the fstrim -v / for that XFS system, which I did daily, I rather received messages like:

8 Gigabytes trimmed.

Now, on top level of the 480GByte Toshiba SSD, I have several Subvolumes like:

# btrfs subvolume list /mnt/rootfs
ID 264 gen 273 top level 5 path @_original_install
ID 265 gen 152 top level 5 path @home_install_ok
ID 266 gen 270 top level 5 path @_snapshot_install_ok             
ID 267 gen 28504 top level 5 path @       
ID 275 gen 28504 top level 5 path @home
ID 276 gen 26900 top level 5 path @Data
ID 607 gen 245 top level 5 path @_snapshot_home_20160330
ID 628 gen 3837 top level 5 path @_root_snapshot_20160402

and when I start the fstrim command, I receive this result:

*****************************************
# fstrim -v /mnt/rootfs/@
  177,3 GiB (190331097088 Bytes) getrimmt
*****************************************

Why is the amount of trimmed space 177 GiB, instead of maybe 8 or 10 like on my old XFS formatted 240 GB SSD?

After trimming my 480GB Toshiba SSD again just after first trim, the result is almost the same, 172 GiB were trimmed now. So: Does fstrim not work for BTRFS?

And, do you know a (very) good tutorial / website or similar, which explains BTRFS, including how subvolume work, what about Meta data?

The more infos on latest btrfs-progs (I use version 4.4.1) the better. If in German, it would be great, too...

And, is it harmful to the SSD, when trimming, or when trimming often?

Answer 1

From the BTRFS Wiki FAQs:

Is Btrfs optimized for SSD?

There are some optimizations for SSD drives, and you can enable them by mounting with -o ssd. As of 2.6.31-rc1, this mount option will be enabled if Btrfs is able to detect non-rotating storage. SSD is going to be a big part of future storage, and the Btrfs developers plan on tuning for it heavily. Note that -o ssd will not enable TRIM/discard.

I see you're not mounting with -o ssd. Perhaps your btrfs-progs isn't detecting it as a SSD. (It checks if /sys/block/sdX/queue/rotational is 0.)

"-o ssd will not enable TRIM/discard" probably because excessive overwrites wear out SSD drives faster.

The fstrim manpage says:

Running fstrim frequently, or even using mount -o discard [which permanently turns TRIM on], might negatively affect the lifetime of poor-quality SSD devices. For most desktop and server systems the sufficient trimming frequency is once a week. Note that not all devices support a queued trim, so each trim command incurs a performance penalty on whatever else might be trying to use the disk at the time.

Also, BTRFS's CoW (copy-on-write) is advantageous for SSDs, minimizing unnecessary overwrites, so TRIM isn't really that necessary with BTRFS. Non-CoW filesystems on SSDs need TRIM turned on.

Why is the amount of trimmed space 177 GiB, instead of maybe 8 or 10 like on my old XFS formatted 240 GB SSD?

Perhaps it's related to ssd_spread not being on, whence you'd have larger, less-fragmented free space:

ssd_spread Mount -o ssd_spread is more strict about finding a large unused region of the disk for new allocations, which tends to fragment the free space more over time. It is often faster on the less expensive SSD devices.

Since your Toshiba Q300 is a lower-end SSD, you should turn on the ssd_spread mount option.

Does fstrim not work for BTRFS?

It does. The BTRFS mount options page says "you can run fstrim command periodically."

And, do you know a (very) good tutorial / website or similar, which explains BTRFS, including how subvolume work, what about Meta data?

This is the best BTRFS resource: https://btrfs.wiki.kernel.org/

The btrfs-subvolume manpage is good. So is the Subvolumes section of the Sysadmin guide. btrfsQuota.py is a neat script for understanding snapshot/subvolume and metadata sizes.

The more infos on latest btrfs-progs (I use version 4.4.1) the better.

The latest is version 4.5.3.

If in German, it would be great, too...

I highly recommend the btrbk Perl script for leveraging BTRFS to do automated backups and snapshoting. It really demonstrates the power of BTRFS. The author, Axel Burri, is from Zurich, Switzerland, and, based upon his having a German first name, he likely knows German, too; perhaps he could point you to some German BTRFS resources.

Also, doing a search on WorldCat, this book mentions BTRFS, but it's somewhat outdated (2011):

• Liebel, Oliver. 2011. Linux Hochverfügbarkeit: Einsatzszenarien und Praxislösungen. [S.l.]: Galileo Press.

Answer 2

Use udev to set your ssd to a non rotational drive.

Edit/add a file called 10-ssd.rules (or whatever) to /etc/udev/rules.d/ and insert a line similar to

ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline" (this sets the ioscheduler for my ssd to mq-deadline, your's will look like something like this

ACTION=="add|change", KERNEL=="sda", ATTR{queue/rotational}=="0" (where sda is your ssd, idk if you can use UIDS, check the manual for the correct settings, you can add ATTR{queue/scheduler}="mq-deadline" too if your kernel supports it since it's faster for ssds and mq and normal ioschedulers are now selectable per device)

I'd use something like rw,lazytime,compress=zstd,ssd,space_cache=v2 in fstab, obviously adjusting for your subvolume config.

If everything worked after a reboot it should say 0 for cat /sys/block/sda/queue/rotational and mq-deadline for cat /sys/block/sda/queue/scheduler.

After that fstrim -va should work on the drive. You shouldn't be issuing discards all the time so it's better to add a systemd timer that runs fstrim every so often.

Answer 3

BTRFS has a much higher overhead, regarding sector writes, than XFS and EXT2/3/4. I wrote a few scripts to test this, you can get them here and see for yourself.

On most of the tests in my scripts, BTRFS writes about 11 times more than XFS (and 48 times more than EXT4). This is most likely due to the "copy on write" way of working of BTRFS.

You are seeing 22 times more bytes trimmed compared to XFS, instead of 11 times, but let's keep in mind my numbers are just tests, and what you're seeing is real world usage.

My advice is to keep BTRFS away from SSDs, or, if you need/want the BTRFS features, keep watching the "wear" SMART attribute on your SSD and be prepared to replace it sooner.