2

I have below IOMMU groups and devices.

$ for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/2/devices/0000:00:04.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:15.0
/sys/kernel/iommu_groups/4/devices/0000:00:15.1
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.7
/sys/kernel/iommu_groups/7/devices/0000:01:00.0
/sys/kernel/iommu_groups/7/devices/0000:02:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.2
/sys/kernel/iommu_groups/8/devices/0000:00:1f.3
/sys/kernel/iommu_groups/8/devices/0000:00:1f.4

I would like to isolate that specific device, /sys/kernel/iommu_groups/7/devices/0000:01:00.0, into its own group with no other devices in the same group.

How can we isolate a single device into separate IOMMU group for PCI passthrough for KVM virtual machine?

0

I know that this is an old question, but I had to try and figure this out recently.

The rule of thumb with IOMMU is that the kernel is going to figure out the mappings for you. When the kernel boots it is going to look for what devices can map to an I/O virtual map (IOVA). If the devices have the same IOVA then they end up in the same group. This is done to guarantee that each group has devices that can actually be addressed and talked to separately.

There are a couple of solutions. The first being that you could try moving the card to another position on the mother board. If it is a PCI and not a PCIe card then you are probably out of luck since all the PCI ports are probably mapped to the same PCIe bridge, and therefore will share the same IOVA.

If you really still need to do it then you can take all the devices that are in the same group and assign them all to vfio-pci and then you can do the assignments afterwards to where they devices need to go.

On my machine for example group 13 has a bunch of devices including an extra video card (18:00.) Here is my output from that directory:

root@rwl01:/sys/kernel/iommu_groups/13/devices# ll
total 0
drwxr-xr-x 2 root root 0 Feb 15 15:43 .
drwxr-xr-x 3 root root 0 Feb 15 15:43 ..
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.1
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:03:00.2 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:16:00.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:00.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:16:01.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:01.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:16:02.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:02.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:16:03.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:03.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:16:04.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:04.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:16:08.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:08.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:17:00.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:00.0/0000:17:00.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:18:00.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:01.0/0000:18:00.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:18:00.1 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:01.0/0000:18:00.1
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:19:00.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:02.0/0000:19:00.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:1a:00.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:03.0/0000:1a:00.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:1b:00.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:04.0/0000:1b:00.0
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:1b:00.1 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:04.0/0000:1b:00.1
lrwxrwxrwx 1 root root 0 Feb 15 15:43 0000:1c:00.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:16:08.0/0000:1c:00.0

As you can see the directory is a bunch of links. Here is the chain on how things are connected:

root@rwl01:/sys/kernel/iommu_groups/13/devices# lspci | grep -E '00:01.3|03:00.2|16:01.0'
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
16:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)

For me all these devices are on the same bridge, my extra video card, raid controller etc. You cannot separate these easily.

BUT.... you can

You will need to apply the https://queuecumber.gitlab.io/linux-acs-override/ (ACS Override Kernel patch) This will allow you to use command line parameters to expose parts of groups are their own groups. After you install the patch you can then configure what you would like kernel command line parameters:

pcie_acs_override =
        [PCIE] Override missing PCIe ACS support for:
    downstream
        All downstream ports - full ACS capabilties
    multifunction
        All multifunction devices - multifunction ACS subset
    id:nnnn:nnnn
        Specfic device - full ACS capabilities
        Specified as vid:did (vendor/device ID) in hex

From here you should be able to make the device be in its own group and you should be off to the races. There are issues with this method:

Here are a couple of good links:

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.