8

I'm creating new setup for my virtual machines and testing which method for storage is most fastest. My test environment consists of HDD drive with LVM on LUKS. I created single LV for virtual machine drive and reused it for both tests to maintain same place on HDD drive to maintain consistent performance (HDD read/write speed depends on physical position).

  • Host: Arch Linux, kernel 4.12.8
  • Guest: Ubuntu Desktop 17.04

Performance tested with command:

    dd if=/dev/zero of=test bs=16M count=100 conv=sync

First test: using LV directly as virtual machine's drive

Command:

qemu-system-x86_64 \
  -drive format=raw,file=/dev/mapper/vg_vm-lv_vm_test,if=virtio,aio=native,cache.direct=on \
  -net nic,model=virtio \
  -net user \
  -vga virtio \
  -display gtk,gl=on \
  -smp 3 \
  -cpu host \
  -machine type=pc,accel=kvm \
  -m 3G

Results (each value represents single run):

  • Creating new file: 98.4 MB/s ; 112 MB/s
  • Writing to existing file: 62.5 MB/s ; 68.7 MB/s ; 64.8 MB /s

Second test: creating ext4 on LV and putting raw image file on that

Command:

qemu-system-x86_64 \
  -drive format=raw,file=./ubuntu_17,if=virtio,aio=native,cache.direct=on \
  -net nic,model=virtio \
  -net user \
  -vga virtio \
  -display gtk,gl=on \
  -smp 3 \
  -cpu host \
  -machine type=pc,accel=kvm \
  -m 3G

Results (each value represents single run):

  • Creating new file: 254 MB/s ; 242 MB/s
  • Writing to existing file: 187 MB/s ; 189 MB/s ; 190 MB/s

Third test: using LV directly as virtual machine's drive, different settings

Command:

qemu-system-x86_64 \
  -drive format=raw,file=/dev/mapper/vg_vm-lv_vm_test,if=virtio,cache=none \
  -net nic,model=virtio \
  -net user \
  -vga virtio \
  -display gtk,gl=on \
  -smp 3 \
  -cpu host \
  -machine type=pc,accel=kvm \
  -m 3G

Results (each value represents single run):

  • Creating new file: 129 MB/s ; 125 MB/s
  • Writing to existing file: 103 MB/s ; 97 MB/s ; 81.9 MB /s

Question

Clearly there is difference between these two solutions however I expected raw block device to be at least as fast as image file because there should be no overhead of host's filesystem. I suppose that some caching occurs in between for file image or options for raw block device are not optimal. Why is raw LV slower in this case? What can I do to improve its performance? Or if it just should be slower, then why?

EDIT: I added third test case using settings from: http://www.linux-kvm.org/page/Tuning_KVM. Turns out to be a bit faster but still slower than file image. I observed also that with each run for existing file it is getting slower - however fragmentation shouldn't occur for file overwrite so I am not sure why it happens.

1 Answer 1

6
+25

Block size mismatch might be the problem.

You ideally want to match your File System with the underlying block size of the media. While you didn't share what you picked for block sizes I think you have 4 kilobytes for Ext4 attempt and 512 bytes for the LV. If your underlying media's natural block size is 4 kilobytes then I think this explains both your speed issue and your diminishing speed. Since you are possibly only writing 512 in to a 4k block you wasted 75% of the block and subsequent writes will use more blocks, incurring more overhead and waste.

Retry your test with a nicely matched File System block size matched with the underlying media block size. In this case try your LV with a 4k block size.

LVM does add a bit more overhead than Ext4. Stack Overflow Answer So I think this would explain why it is just a bit slower. :)

3
  • 1
    You were right with block sizes! It did improve block device performance however it is still a bit slower than file - writing is about 20% slower and reading 5% slower. Any other ideas? Sep 12, 2017 at 20:00
  • Hello :) At this point I'd verify what sizes you have for the underlying hardware to be sure you know you're working with a 4K block size,it could be 8k or 16k or something else. So how you can do that is first: sudo df then find your mount point. Mine is / for example. Notice to the left it will have the associated block device. In my case it is /dev/sda1. Then run this command, blockdev --getbsz /dev/sda1 Notice the size of your block in bytes. Now you know for sure what size you should match your images to. Sep 13, 2017 at 12:50
  • Block size is 4k for this device (and it is correct for real device, logical volume and on virtual machine). Sep 14, 2017 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.