4

I have searched via google but did not find any information on the read balancing algorithm for MD raid1.

Some years ago i have maybe read somewhere that it uses hashing the process id to determine which device it uses, but i am not certain.

I have KVM virtualisation + LVM on top of raid1 and i witnessed multiple times that reads from multiple virtual machines go to the same disk, going 100% utilization and the other disk in the raid1 is idle. (via iostat -x)

EDIT: the question is also about multiple processes inside the VM. Will the read balancing happen even if inside the VM?

Are the reads balanced? If yes by which algorithm?

The reason i am asking is because i am currently dedicing between two variants

  1. LUKS on top of single MD device
  2. MD on top of two LUKS devices

The second variant uses more CPU ,but that is not a concern.

Improve this question

2 Answers 2

Reset to default
4

Linux implementation of RAID1 speeds up disk read operations as long as two separate disk read operations are performed at a time. That means reading one file won't be any faster on RAID1 than on single disk, but reading two distinct files in the same time will be faster.

Read test done with DD and read cache disabled:

Test single file:
1048576000 copied @ 224MB/s

Test same file 2 transfers:
Test single file:
1048576000 copied @ 116MB/s
1048576000 copied @ 104MB/s

Test 2 files 2 transfers:
1048576000 copied @ 212MB/s
1048576000 copied @ 217MB/s

As for the options, LUKS on top of single MD device sounds more logical.

The problem with your reads that happen from the same disk could be tweaked with best_dist_disk and best_pending_disk parameters. You can see a complete example here.

Improve this answer
2
  • have you tried this read test inside a VM on top of raid1 ?
    – Sidias-Korrado
    Jan 20, 2017 at 9:18
  • No, the test was on a physical machine directly. In the case of VM it will not work because the VM is a single file or a number of very large files and simultaneous reading is not therefore possible for a single VM file (or highly improbable in the case of multiple files). Allocating disks and making the RAID directly from the VM if possible would solve this problem, since the VM would have direct physical RAID access.
    – Overmind
    Jan 20, 2017 at 9:27
2

If you only have a single stream of sequential I/O the md RAID1 algorithm will keep picking the same disk. From the mdadm man page:

[On md RAID1] a single stream of sequential input will not be accelerated (e.g. a single dd), but multiple sequential streams or a random workload will use more than one spindle. In theory, having an N-disk RAID1 will allow N sequential threads to read from all disks.

You can read the source code for the 5.10 kernel to see the md RAID1 balancing algorithm. The rough overview is:

  • Balancing can only happen in regions where there are multiple disks that are in-sync and are not faulty
  • Balancing will try and avoid disks marked as "write mostly"
  • If a disk's last I/O finished exactly before the area that wants to be read, balancing will continue with that disk unless the new I/O's size is too big (this is why a single sequential stream is not accelerated)
  • If the above didn't give us a disk, check if any of the disks is an SSD or if any of the disks has no pending I/O. If either of these cases is true, pick the disk with the least amount of pending I/O.
  • If the above didn't give us disk, pick the disk that most recently did I/O closest to the desired read's location
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.