Bad performance with Linux software RAID5 and LUKS encryption

Question

I have set up a Linux software RAID5 on three hard drives and want to encrypt it with cryptsetup/LUKS. My tests showed that the encryption leads to a massive performance decrease that I cannot explain.

The RAID5 is able to write 187 MB/s [1] without encryption. With encryption on top of it, write speed is down to about 40 MB/s.

The RAID has a chunk size of 512K and a write intent bitmap. I used -c aes-xts-plain -s 512 --align-payload=2048 as the parameters for cryptsetup luksFormat, so the payload should be aligned to 2048 blocks of 512 bytes (i.e., 1MB). cryptsetup luksDump shows a payload offset of 4096. So I think the alignment is correct and fits to the RAID chunk size.

The CPU is not the bottleneck, as it has hardware support for AES (aesni_intel). If I write on another drive (an SSD with LVM) that is also encrypted, I do have a write speed of 150 MB/s. top shows that the CPU usage is indeed very low, only the RAID5 xor takes 14%.

I also tried putting a filesystem (ext4) directly on the unencrypted RAID so see if the layering is problem. The filesystem decreases the performance a little bit as expected, but by far not that much (write speed varying, but > 100 MB/s).

Summary:
Disks + RAID5: good
Disks + RAID5 + ext4: good
Disks + RAID5 + encryption: bad
SSD + encryption + LVM + ext4: good

The read performance is not affected by the encryption, it is 207 MB/s without and 205 MB/s with encryption (also showing that CPU power is not the problem).

What can I do to improve the write performance of the encrypted RAID?

[1] All speed measurements were done with several runs of dd if=/dev/zero of=DEV bs=100M count=100 (i.e., writing 10G in blocks of 100M).

Edit: If this helps: I'm using Ubuntu 11.04 64bit with Linux 2.6.38.

Edit2: The performance stays approximately the same if I pass a block size of 4KB, 1MB or 10MB to dd.

Answer 1

LUKS has a botleneck, that is it just spawns one thread per block device.

Are you placing the encryption on top of the RAID 5? Then from the point of view of your OS you just have one device, then it is using just one thread for all those disks, meaning disks are working in a serial way rather than parallel.

Are you placing LVM on top of the encrypted devices? Then you are spawning one thread per device allowing them to work in parallel.

A solution for that? If you are using Linux, regardless your CPU speed, power or core number, use software raid, then create the volumes on top of encrypted disks. That way your OS will see multiple devices and spawn a thread per each one.

The downside is to have to enter the passkey many times, but to avoid that you can go for this setup. One disk or portion of a disk have the boot and root partitions, boot is unencrypted, root is encrypted and holds a keyfile, that keyfile is used to decrypt the other devices (maybe the rest of the same disk).

I found this problem not using RAID, but dealing with a virtual machine that was slowing down the host OS, until i realized the gues OS was monopolizing the luks thread i decided to create different partitions on the same device and use one partition for the virtual disks, then they cannot monopolize the thread and both OSes performance improved a lot.

I think you can also benefit of using software RAID, you'd end not dependant on your hardware.