1

Summary

I'm trying to set-up GPU-passthrough on a linux host with KVM so that I might run a Windows guest for gaming. I've been having a problem where some IOMMU entries (all belonging to the GPU I wish to passthrough) are being assigned to the vfio bus, while others are assigned to the nvidia-gpu drivers.

Hardware

CPU - Threadripper 1950X

GPU - Sapphire RX 480 (AMD, meant for host graphics) - MSI Duke RTX 2080 (nVidia, meant for guest graphics)

MoBo - MSI Carbon Gaming Pro AC x399

Problem

When I try to passthrough my nVidia GPU, I get the error Please ensure all devices within the iommu_group are bound to their vfio bus driver. (see images below).

When I run lspci -k, I find the following records:

0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GV104 [GeForce GTX 1180] [10de:1e87] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3721]
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau
0b:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f8] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3721]
    Kernel driver in use: vfio-pci
    Kernel modules: snd_hda_intel
0b:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ad8] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3721]
    Kernel driver in use: xhci_hcd
0b:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad9] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3721]
    Kernel driver in use: nvidia-gpu
    Kernel modules: i2c_nvidia_gpu
41:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X] [1002:67df] (rev c7)
    Subsystem: PC Partner Limited / Sapphire Technology Radeon RX 470/480 [174b:e347]
    Kernel driver in use: amdgpu
    Kernel modules: amdgpu
41:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
    Subsystem: PC Partner Limited / Sapphire Technology Ellesmere [Radeon RX 580] [174b:aaf0]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel

Which reports that the RTX 2080 is being used by vfio-pci on 0b:00.0, and 0b:00.1, but 0b:00.2 and 0b.00.3 (which all belong to the RTX 2080) are being used by xhci_hcd and nvidia-gpu.

My understanding is that this is the issue.

What I've done

I mostly referred a Level1 thread and a blog post by Jack Ford:

Blacklisted Nouveau

in file /etc/modprobe.d/blacklist-nouveau.conf

blacklist nouveau
options nouveau modeset=0

Updated GRUB

In /etc/default/grub:

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=on vfio-pci.ids=10de:1e87,10de:10f8,10de:1ad8,10de:1ad9"
GRUB_CMDLINE_LINUX=""

Added kernel modules

In /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

In /etc/modprobe.d/vfio.conf:

options vfio-pci ids=10de:1e87,10de:10f8,10de:1ad8,10de:1ad9

In /etc/modules-load.d/vfio-pci.conf:

vfio-pci

Confirmed IOMMU/VFIO is enabled

$ dmesg | grep -E "DMAR|IOMMU"
[    0.946794] AMD-Vi: IOMMU performance counters supported
[    0.946855] AMD-Vi: IOMMU performance counters supported
[    0.976135] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    0.976139] AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40
[    0.977856] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    0.977873] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[    1.367168] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
$ dmesg | grep -i vfio
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-37-generic root=UUID=27ff4f46-db04-4ca0-8981-191a45c365ad ro quiet splash amd_iommu=on vfio-pci.ids=10de:1e87,10de:10f8,10de:1ad8,10de:1ad9 vt.handoff=1
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-37-generic root=UUID=27ff4f46-db04-4ca0-8981-191a45c365ad ro quiet splash amd_iommu=on vfio-pci.ids=10de:1e87,10de:10f8,10de:1ad8,10de:1ad9 vt.handoff=1
[    2.667765] VFIO - User Level meta-driver version: 0.3
[    2.675583] vfio-pci 0000:0b:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    2.692215] vfio_pci: add [10de:1e87[ffffffff:ffffffff]] class 0x000000/00000000
[    2.712193] vfio_pci: add [10de:10f8[ffffffff:ffffffff]] class 0x000000/00000000
[    2.712199] vfio_pci: add [10de:1ad8[ffffffff:ffffffff]] class 0x000000/00000000
[    2.712204] vfio_pci: add [10de:1ad9[ffffffff:ffffffff]] class 0x000000/00000000
[    4.960496] vfio-pci 0000:0b:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

Conclusion

Thanks so much for helping out... I don't know what those two other entries are in the nVidia IOMMU group, but they all appear to belong to the nVidia device. I'm guessing if I could disable the USB and Serial controller, things might work?

2
  • As blacklisting a driver isn't exactly a "proper" way to do passthrough. The best way is to make sure that the vfio-pci is loaded before all other drivers and hence will be able to bind all the devices it needs to. How to achieve that often depends on the init in the initramfs. (an appropriate pre in modprobe conf could be an alternative as well). – Tom Yan Dec 17 '19 at 5:47
  • Does pass through really work for now? Does Windows host +Windows guest work for gaming? – jw_ Mar 11 '20 at 4:16
0

Recently facing the same issue. The main objective is to have wifi use vfio, but need to move a USB and a RAM also, as they are in the same IOMMU group.

I did all the steps stated in questions, got RAM and WIFI device on vfio, but not USB.

Finally found out that the xhci_hcd is compiled into the kernel, not as module. So putting the USB controller id in GRUB_CMDLINE_LINUX_DEFAULT, the USB controller will still bind with xhci_hcd, as it is already inside the kernel. To confirm if your kernel compile xhci_hcd within:

modprobe -D xhci_hcd

If your output is:

builtin xhci_hcd

Then your kernel contain xhci_hcd.

I end up with a script to do the manual driver unbind and bind:

#!/bin/sh
PCI_HID=
echo -n "$PCI_HID" > /sys/bus/pci/drivers/xhci_hcd/unbind
echo -n "$PCI_HID" > /sys/bus/pci/drivers/vfio-pci/bind

For the question. Following should work:

#!/bin/sh
PCI_HID="0000:0b:00.2"
echo -n "$PCI_HID" > /sys/bus/pci/drivers/xhci_hcd/unbind
echo -n "$PCI_HID" > /sys/bus/pci/drivers/vfio-pci/bind
PCI_HID="0000:0b:00.3"
echo -n "$PCI_HID" > /sys/bus/pci/drivers/nvidia-gpu/unbind
echo -n "$PCI_HID" > /sys/bus/pci/drivers/vfio-pci/bind

PS: Though we are manually binding and unbinding drivers, the GRUB_CMDLINE_LINUX_DEFAULT and module config steps are still necessary, else vfio will not bind to the USB controller.

0

I know I am a bit late to the party but I hope the following can help people facing similar issues. Please note I have only tested this on discrete USB controllers.

xhci_hcd is indeed built in the kernel so it claims the USB controller and unless a workaround is found such as a script to bind/unbind vfio-pci is not assigned as the kernel in use.

Another way to solve this issue is to use the following command:

sudo driverctl -v set-override 0000:0X:00 vfio-pci

Where 0000:0X:00 is your pci id.

Upon reboot the driver assigned will be automatically be vfio-pci.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.