I have set up a Linux software RAID5 on three hard drives and want to encrypt it with cryptsetup/LUKS. My tests showed that the encryption leads to a massive performance decrease that I cannot explain.

The RAID5 is able to write 187 MB/s [1] without encryption. With encryption on top of it, write speed is down to about 40 MB/s.

The RAID has a chunk size of 512K and a write intent bitmap. I used -c aes-xts-plain -s 512 --align-payload=2048 as the parameters for cryptsetup luksFormat, so the payload should be aligned to 2048 blocks of 512 bytes (i.e., 1MB). cryptsetup luksDump shows a payload offset of 4096. So I think the alignment is correct and fits to the RAID chunk size.

The CPU is not the bottleneck, as it has hardware support for AES (aesni_intel). If I write on another drive (an SSD with LVM) that is also encrypted, I do have a write speed of 150 MB/s. top shows that the CPU usage is indeed very low, only the RAID5 xor takes 14%.

I also tried putting a filesystem (ext4) directly on the unencrypted RAID so see if the layering is problem. The filesystem decreases the performance a little bit as expected, but by far not that much (write speed varying, but > 100 MB/s).

Disks + RAID5: good
Disks + RAID5 + ext4: good
Disks + RAID5 + encryption: bad
SSD + encryption + LVM + ext4: good

The read performance is not affected by the encryption, it is 207 MB/s without and 205 MB/s with encryption (also showing that CPU power is not the problem).

What can I do to improve the write performance of the encrypted RAID?

[1] All speed measurements were done with several runs of dd if=/dev/zero of=DEV bs=100M count=100 (i.e., writing 10G in blocks of 100M).

Edit: If this helps: I'm using Ubuntu 11.04 64bit with Linux 2.6.38.

Edit2: The performance stays approximately the same if I pass a block size of 4KB, 1MB or 10MB to dd.

  • 1
    I encountered a similar problem several years ago, and ended up adding a disk and switching to RAID-10. That made a noticeable difference, though it's still not as fast as I'd like since that machine lacks AES-NI. – Wyzard Jul 3 '11 at 14:38
  • @Wyzard That might be a solution, but I'd like to avoid it as it needs more hard disks, especially as I plan to increase the size of the RAID by adding another disk in the future (RAID-10 would then already need 6 disks, which do not fit in my machine). – Philipp Wendler Jul 5 '11 at 6:25
  • Software RAID5 is horribly slow, because it has to calculate parity bits, which can't meaningfully be accelerated on the CPU. A dedicated RAID processor in a hardware RAID controller can offload the parity bit calculations from the CPU, and the RAID controller's APU is designed to perform well enough to handle the max throughput of the port. Even if the CPU cost of your encryption were 0, the bottleneck is likely within the RAID5 stack. Software RAID5 is just a poor idea in my opinion, as 99.999% of the solutions that prescribe it would be better served by hardware RAID or JBOD. – allquixotic Aug 30 '12 at 15:09
  • @allquixotic No, that's not true. Parity bit calculation is quite cheap and no problem for today's cpus. My cpu can handle the parity bit calculation for several 13 GB per second (it benchmarks this on boot and prints the result to syslog). Also the benchmark results posted in the question show that the RAID is not the problem, because RAID without encryption is fast, as is encryption without RAID. Only their combination causes the problem. – Philipp Wendler Aug 30 '12 at 20:45

The solution is to set the stripe_cache_size feature for md raids.

By default it is set to 256, but it can be increased up to 32768.

This is done by writing the desired size to /sys/block/md0/md/stripe_cache_size (if the raid is md0). On Ask Ubuntu there is a solution for setting the value permanently.

I tested on the exact same RAID as in the question, and I got the following numbers:

size   256: 50 MB/s
size  4096: 123 MB/s
size  8192: 142 MB/s
size 16384: 140 MB/s
size 32768: 142 MB/s

These tests were conducted with Ubuntu 12.04 (Linux 3.2) by writing 10 GB into a file with blocks of 1 MB.

Background: The stripe cache stores recently written blocks. If data is written continuously, it might happen that during a first write only a part of one stripe is written. This means, the RAID code has to read the complete stripe from disk, update it, and write it completely again. If a second write comes in for another part of the same stripe, all this would have to be done again. Now, if the cache is used and still contains the data written by the first write, the read that was necessary before the second write can be omitted.

Usually a big block size when writing would prevent the problem (because full stripes are written at once, and thus no reads at all are necessary). However, it seems that the encryption uses only small blocks when writing to the underlying device, and thus increasing the cache has a positive effect.

| improve this answer | |
  • does anyone know how to do this when using LVM to do the raid5 and on a newer kernel? My system has no /sys/block/mdN, only /sys/block/dm-M and there is no stripe_cache_size anywhere in it. – Jay R. Wren Sep 9 '19 at 21:20

LUKS has a botleneck, that is it just spawns one thread per block device.

Are you placing the encryption on top of the RAID 5? Then from the point of view of your OS you just have one device, then it is using just one thread for all those disks, meaning disks are working in a serial way rather than parallel.

Are you placing LVM on top of the encrypted devices? Then you are spawning one thread per device allowing them to work in parallel.

A solution for that? If you are using Linux, regardless your CPU speed, power or core number, use software raid, then create the volumes on top of encrypted disks. That way your OS will see multiple devices and spawn a thread per each one.

The downside is to have to enter the passkey many times, but to avoid that you can go for this setup. One disk or portion of a disk have the boot and root partitions, boot is unencrypted, root is encrypted and holds a keyfile, that keyfile is used to decrypt the other devices (maybe the rest of the same disk).

I found this problem not using RAID, but dealing with a virtual machine that was slowing down the host OS, until i realized the gues OS was monopolizing the luks thread i decided to create different partitions on the same device and use one partition for the virtual disks, then they cannot monopolize the thread and both OSes performance improved a lot.

I think you can also benefit of using software RAID, you'd end not dependant on your hardware.

| improve this answer | |
  • 1
    First, according to this commit (which was integrated into Linux 2.6.38), dm-crypt is now at least in some situations multi-threaded. This doesn't apply in my artificial benchmark situation as there is only one writer, but should speed up normal operation. Second, I don't think what you describe is correct even before that enhancement. Previously there would be a single thread per DM-Crypt device, so putting a LVM inside that would not enhance the number of threads. – Philipp Wendler Aug 15 '11 at 10:39
  • Third, this doesn't explain my problem here. I have two different situations: a single encrypted volume on my SSD, and a single encrypted volume inside my RAID5. One is fast, the other is slow. The number of threads used is exactly the same in both cases. – Philipp Wendler Aug 15 '11 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.