Okay folks! What needs to known in advance is that my initial VM solution has a working installation with a macOS Ventura guest on a Kali/Debian Linux host prior to extending the baseline configuration to further incorporate single GPU passthrough as a means for much-needed improved performance.
Using the latest GNOME with Kali (Debian-based) Linux featuring a custom compiled 6.2.2 kernel.
Hardware is a MacBook Air 2013 with Haswell CPU and Intel i915 GPU (with HDMI).
$ uname -a
Linux ... 6.2.2 #1 SMP PREEMPT Thu Mar 9 20:05:06 GMT 2023 x86_64 GNU/Linux
$ cat /proc/cpuinfo
...
model name : Intel(R) Core(TM) i5-4250U CPU @ 1.30GHz
...
First and foremost, the following is the relevant GRUB 2 configuration:
$ grep LINUX_DEFAULT /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="nosplash loglevel=3 intel_iommu=on i915.enable_gvt=1 i915.enable_guc=0 vfio-pci.ids=8086:0a26,8086:0a0c video=efifb:off kvm.ignore_msrs iommu=pt”
Also, the following verifies IOMMU configuration accordingly:
$ dmesg | grep -i iommu
[ 0.000000] Command line: BOOT_IMAGE=/BOOT/sunv2@/vmlinuz-6.2.2 root=ZFS=/ROOT/sunv2 ro root=ZFS=zroot/ROOT/sunv2 nosplash loglevel=3 intel_iommu=on i915.enable_gvt=1 i915.enable_guc=0 vfio-pci.ids=8086:0a26,8086:0a0c video=efifb:off kvm.ignore_msrs iommu=pt
[ 0.038172] Kernel command line: BOOT_IMAGE=/BOOT/sunv2@/vmlinuz-6.2.2 root=ZFS=/ROOT/sunv2 ro root=ZFS=zroot/ROOT/sunv2 nosplash loglevel=3 intel_iommu=on i915.enable_gvt=1 i915.enable_guc=0 vfio-pci.ids=8086:0a26,8086:0a0c video=efifb:off kvm.ignore_msrs iommu=pt
[ 0.038366] DMAR: IOMMU enabled
[ 0.113621] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.255936] iommu: Default domain type: Passthrough (set via kernel command line)
[ 0.293253] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.293257] DMAR: IOMMU feature sc_support inconsistent
[ 0.293259] DMAR: IOMMU feature pass_through inconsistent
[ 0.340122] pci 0000:00:02.0: Adding to iommu group 0
[ 0.340249] pci 0000:00:00.0: Adding to iommu group 1
[ 0.340278] pci 0000:00:03.0: Adding to iommu group 2
[ 0.340305] pci 0000:00:14.0: Adding to iommu group 3
[ 0.340349] pci 0000:00:15.0: Adding to iommu group 4
[ 0.340372] pci 0000:00:15.4: Adding to iommu group 4
[ 0.340404] pci 0000:00:16.0: Adding to iommu group 5
[ 0.340430] pci 0000:00:1b.0: Adding to iommu group 6
[ 0.340456] pci 0000:00:1c.0: Adding to iommu group 7
[ 0.340480] pci 0000:00:1c.1: Adding to iommu group 8
[ 0.340506] pci 0000:00:1c.2: Adding to iommu group 9
[ 0.340532] pci 0000:00:1c.4: Adding to iommu group 10
[ 0.340557] pci 0000:00:1c.5: Adding to iommu group 11
[ 0.340599] pci 0000:00:1f.0: Adding to iommu group 12
[ 0.340624] pci 0000:00:1f.3: Adding to iommu group 12
[ 0.340648] pci 0000:02:00.0: Adding to iommu group 13
[ 0.340678] pci 0000:03:00.0: Adding to iommu group 14
[ 0.340703] pci 0000:05:00.0: Adding to iommu group 15
[ 0.340731] pci 0000:06:00.0: Adding to iommu group 16
[ 0.340757] pci 0000:06:03.0: Adding to iommu group 17
[ 0.340784] pci 0000:06:04.0: Adding to iommu group 18
[ 0.340809] pci 0000:06:05.0: Adding to iommu group 19
[ 0.340834] pci 0000:06:06.0: Adding to iommu group 20
[ 0.340845] pci 0000:07:00.0: Adding to iommu group 16
[ 0.340871] pci 0000:04:00.0: Adding to iommu group 21
The following is the relevant log output (note no errors or failure) when starting the VM from virt-manager:
$ sudo tail -f /var/log/libvirt/qemu/macOS.log
2023-03-10 14:01:12.580+0000: starting up libvirt version: 9.0.0, package: 1 (Andrea Bolognani <[email protected]> Sat, 28 Jan 2023 17:03:53 +0100), qemu version: 7.2.0Debian 1:7.2+dfsg-4, kernel: 6.2.2, hostname: macbook-spellegrino.sunv2.com
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-1-macOS \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-macOS/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-macOS/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-macOS/.config \
/usr/bin/qemu-system-x86_64 \
-name guest=macOS,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-macOS/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/vm/macOS/git/OSX-KVM/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/vm/macOS/git/OSX-KVM/OVMF_VARS-1024x768.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-4.2,usb=off,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-accel kvm \
-cpu qemu64 \
-m 2048 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":2147483648}' \
-overcommit mem-lock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-uuid 2aca0dd6-cec9-4717-9ab2-0b7b13d111c3 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=33,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"}' \
-device '{"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"}' \
-device '{"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"}' \
-device '{"driver":"pcie-root-port","port":14,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x1.0x6"}' \
-device '{"driver":"pcie-root-port","port":15,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x1.0x7"}' \
-device '{"driver":"pcie-pci-bridge","id":"pci.9","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pcie.0","addr":"0x7.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pcie.0","multifunction":true,"addr":"0x7"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pcie.0","addr":"0x7.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pcie.0","addr":"0x7.0x2"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/vm/macOS/git/OSX-KVM/OpenCore/OpenCore.qcow2","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
-device '{"driver":"ide-hd","bus":"ide.0","drive":"libvirt-2-format","id":"sata0-0-0","bootindex":2,"write-cache":"on"}' \
-blockdev '{"driver":"file","filename":"/vm/macOS/macOS.qcow2","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"ide-hd","bus":"ide.1","drive":"libvirt-1-format","id":"sata0-0-1","bootindex":1,"write-cache":"on"}' \
-netdev '{"type":"tap","fd":"34","id":"hostnet0"}' \
-device '{"driver":"vmxnet3","netdev":"hostnet0","id":"net0","mac":"52:54:00:e6:85:40","bus":"pci.9","multifunction":true,"addr":"0x1"}' \
-chardev socket,id=charchannel0,fd=32,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"vfio-pci","host":"0000:00:02.0","id":"hostdev0","bus":"pcie.0","multifunction":true,"addr":"0x3.0x1","rombar":1,"romfile":"/vm/macOS/i915/i915vbios.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:00:03.0","id":"hostdev1","bus":"pcie.0","addr":"0x3.0x3","rombar":1}' \
-fw_cfg name=opt/igd-opregion,file=/vm/macOS/i915/opregion.bin \
-fw_cfg name=opt/igd-bdsm-size,file=/vm/macOS/i915/bdsmSize.bin \
-device 'isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc' \
-smbios type=2 \
-usb \
-device usb-tablet \
-device usb-kbd \
-cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check \
-sandbox off \
-msg timestamp=on
In order to isolate the problem the following relaxation of privileges has been applied (until a fully working solution):
$ sudo grep -v \# /etc/libvirt/qemu.conf
user = "+0"
group = "+0"
seccomp_sandbox = 0
$ id
...27(sudo),29(audio),30(dip),44(video),46(plugdev),102(input),104(kvm),106(netdev),120(libvirt)
$ sudo grep -v \# /etc/security/limits.conf
@kvm soft memlock unlimited
@kvm hard memlock unlimited
@libvirt soft memlock unlimited
@libvirt hard memlock unlimited
$ sudo cat /etc/udev/rules.d/vfio-kvm.rules
SUBSYSTEM=="vfio", OWNER="root", GROUP=“kvm"
$ sudo udevadm control —reload
$ sudo udevadm trigger
In terms of hooks before/after VM running is the following:
$ tree /etc/libvirt/hooks
/etc/libvirt/hooks
├── qemu
└── qemu.d
└── macOS
├── prepare
│ └── begin
│ └── start.sh
└── release
└── end
└── stop.sh
7 directories, 3 files
$ cat /etc/libvirt/hooks/qemu
#!/bin/bash
GUEST_NAME="$1"
HOOK_NAME="$2"
STATE_NAME="$3"
MISC="${@:4}"
BASEDIR="$(dirname $0)"
HOOKPATH="$BASEDIR/qemu.d/$GUEST_NAME/$HOOK_NAME/$STATE_NAME"
set -e # If a script exits with an error, we should as well.
if [ -f "$HOOKPATH" ]; then
eval \""$HOOKPATH"\" "$@"
elif [ -d "$HOOKPATH" ]; then
while read file; do
eval \""$file"\" "$@"
done <<< "$(find -L "$HOOKPATH" -maxdepth 1 -type f -executable -print;)"
fi
$ sudo cat /etc/libvirt/hooks/qemu.d/macOS/prepare/begin/start.sh
#!/bin/bash
set -x
# Stop display manager
systemctl stop display-manager
# Unbind VTconsoles
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
# Unbind EFI Framebuffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
sleep 3
modprobe --remove-dependencies -f i915
modprobe --remove-dependencies -f snd_hda_intel
sleep 3
# Unload AMD kernel module
# modprobe -r amdgpu
# Detach GPU devices from host
# Use your GPU and HDMI Audio PCI host device
virsh nodedev-detach pci_0000_00_02_0
virsh nodedev-detach pci_0000_00_03_0
sleep 3
# Load vfio module
modprobe vfio-pci
$ cat /etc/libvirt/hooks/qemu.d/macOS/release/end/stop.sh
#!/usr/bin/bash
set -x
# Attach GPU devices to host
# Use your GPU and HDMI Audio PCI host device
virsh nodedev-reattach pci_0000_00_02_0
virsh nodedev-reattach pci_0000_00_03_0
# Unload vfio module
modprobe -r vfio-pci
# Rebind framebuffer to host
echo "efi-framebuffer.0" > /sys/bus/platform/drivers/efi-framebuffer/bind
modprobe i915
modprobe snd_hda_intel
# Bind VTconsoles
echo 1 > /sys/class/vtconsole/vtcon0/bind
echo 1 > /sys/class/vtconsole/vtcon1/bind
# Restart Display Manager
systemctl start display-manager
These hook scripts when executed produce the following output (note that there seems to be an issue when unbinding from the EFI framebuffer with echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind):
$ sudo /etc/libvirt/hooks/qemu macOS prepare begin
+ systemctl stop display-manager
+ echo 0
+ echo 0
+ echo efi-framebuffer.0
/etc/libvirt/hooks/qemu.d/macOS/prepare/begin/start.sh: line 12: echo: write error: No such device
+ sleep 3
+ modprobe --remove-dependencies -f i915
+ modprobe --remove-dependencies -f snd_hda_intel
+ sleep 3
+ virsh nodedev-detach pci_0000_00_02_0
Device pci_0000_00_02_0 detached
+ virsh nodedev-detach pci_0000_00_03_0
Device pci_0000_00_03_0 detached
+ sleep 3
+ modprobe vfio-pci
+ sleep 3
Notice that the following VFIO devices are available upon execution of the hook start.sh script:
$ ls -l /dev/vfio
total 0
crw-rw---- 1 root kvm 244, 0 Mar 10 14:32 0
crw-rw---- 1 root kvm 244, 1 Mar 10 14:32 2
crw-rw-rw- 1 root root 10, 196 Mar 10 14:08 vfio
There appears to be no issues with the hook end.sh script, which brings the system back to a known consistent state for the host OS:
$ sudo /etc/libvirt/hooks/qemu macOS release end
+ virsh nodedev-reattach pci_0000_00_02_0
Device pci_0000_00_02_0 re-attached
+ virsh nodedev-reattach pci_0000_00_03_0
Device pci_0000_00_03_0 re-attached
+ modprobe -r vfio-pci
+ echo efi-framebuffer.0
/etc/libvirt/hooks/qemu.d/macOS/release/end/stop.sh: line 16: echo: write error: No such device
+ modprobe i915
+ modprobe snd_hda_intel
+ echo 1
+ echo 1
+ systemctl start display-manager
Notice that the previous VFIO devices are now unavailable upon execution of the hook end.sh script:
$ ls -l /dev/vfio
total 0
crw-rw-rw- 1 root root 10, 196 Mar 10 14:08 vfio
Also, please make a reference to the “Kernel driver in use” from the following output (before invoking VM), which in this case is i915 and snd_hda_intel:
$ lspci -v
...
00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated Graphics Controller (rev 09) (prog-if 00 [VGA controller])
Subsystem: Apple Inc. Haswell-ULT Integrated Graphics Controller
Flags: bus master, fast devsel, latency 0, IRQ 70, NUMA node 0, IOMMU group 0
Memory at b0000000 (64-bit, non-prefetchable) [size=4M]
Memory at a0000000 (64-bit, prefetchable) [size=256M]
I/O ports at 2000 [size=64]
Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: i915
Kernel modules: i915
00:03.0 Audio device: Intel Corporation Haswell-ULT HD Audio Controller (rev 09)
Subsystem: Apple Inc. Haswell-ULT HD Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 75, NUMA node 0, IOMMU group 2
Memory at b0a10000 (64-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
Furthermore, please make another reference to the “Kernel driver in use” from the following output (after invoking VM), which in this case is now vfio-pci for both the previous i915 and snd_hda_intel instances (therefore we are bound to the right driver for GPU/PCI passthrough):
00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated Graphics Controller (rev 09) (prog-if 00 [VGA controller])
Subsystem: Apple Inc. Haswell-ULT Integrated Graphics Controller
Flags: bus master, fast devsel, latency 0, IRQ 16, NUMA node 0, IOMMU group 0
Memory at b0000000 (64-bit, non-prefetchable) [size=4M]
Memory at a0000000 (64-bit, prefetchable) [size=256M]
I/O ports at 2000 [size=64]
Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: i915
00:03.0 Audio device: Intel Corporation Haswell-ULT HD Audio Controller (rev 09)
Subsystem: Apple Inc. Haswell-ULT HD Audio Controller
Flags: fast devsel, IRQ 16, NUMA node 0, IOMMU group 2
Memory at b0a10000 (64-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
For brevity, there is even more output to reference in relation to troubleshooting this problem:
$ cat /etc/modules-load.d/kvm.conf
kvmgt
vfio-iommu-type1
mdev
vfio-pci
$ lsmod | grep -iE '(vfio|iommu|mdev|i915|snd_hda_intel)'
vfio_pci 24576 0
vfio_pci_core 131072 1 vfio_pci
snd_hda_intel 73728 1
snd_intel_dspcfg 40960 1 snd_hda_intel
snd_hda_codec 278528 3 snd_hda_codec_generic,snd_hda_intel,snd_hda_codec_cirrus
snd_hda_core 180224 4 snd_hda_codec_generic,snd_hda_intel,snd_hda_codec,snd_hda_codec_cirrus
snd_pcm 258048 3 snd_hda_intel,snd_hda_codec,snd_hda_core
snd 200704 12 snd_hda_codec_generic,snd_seq,snd_seq_device,snd_hwdep,snd_hda_intel,snd_hda_codec,snd_timer,snd_pcm
mdev 28672 1 kvmgt
vfio_iommu_type1 73728 0
vfio 86016 4 vfio_pci_core,kvmgt,vfio_iommu_type1,vfio_pci
i915 5607424 11 kvmgt
drm_buddy 28672 1 i915
i2c_algo_bit 24576 1 i915
drm_display_helper 274432 1 i915
cec 98304 2 drm_display_helper,i915
ttm 151552 1 i915
drm_kms_helper 323584 2 drm_display_helper,i915
drm 974848 10 drm_kms_helper,kvmgt,drm_display_helper,drm_buddy,i915,ttm
video 81920 1 i915
Something worth mentioning is that there are NO mdev_supported_types for the underlying device from the output that follows (it would be good if I could get more insight and information regarding this):
$ ls -l /sys/devices/pci0000:00/0000:00:02.0/
total 0
-r--r--r-- 1 root root 4096 Mar 10 14:09 ari_enabled
drwxr-xr-x 3 root root 0 Mar 10 14:20 backlight
-r--r--r-- 1 root root 4096 Mar 10 13:59 boot_vga
-rw-r--r-- 1 root root 4096 Mar 10 14:09 broken_parity_status
-r--r--r-- 1 root root 4096 Mar 10 13:59 class
-rw-r--r-- 1 root root 256 Mar 10 13:59 config
-r--r--r-- 1 root root 4096 Mar 10 14:09 consistent_dma_mask_bits
lrwxrwxrwx 1 root root 0 Mar 10 14:20 consumer:pci:0000:00:03.0 -> ../../virtual/devlink/pci:0000:00:02.0--pci:0000:00:03.0
-rw-r--r-- 1 root root 4096 Mar 10 14:09 d3cold_allowed
-r--r--r-- 1 root root 4096 Mar 10 13:58 device
-r--r--r-- 1 root root 4096 Mar 10 14:09 dma_mask_bits
lrwxrwxrwx 1 root root 0 Mar 10 14:20 driver -> ../../../bus/pci/drivers/i915
-rw-r--r-- 1 root root 4096 Mar 10 14:20 driver_override
drwxr-xr-x 4 root root 0 Mar 10 14:20 drm
-rw-r--r-- 1 root root 4096 Mar 10 14:09 enable
lrwxrwxrwx 1 root root 0 Mar 10 14:09 firmware_node -> ../../LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00
drwxr-xr-x 3 root root 0 Mar 10 14:20 graphics
drwxr-xr-x 3 root root 0 Mar 10 14:20 i2c-1
drwxr-xr-x 3 root root 0 Mar 10 14:20 i2c-2
drwxr-xr-x 3 root root 0 Mar 10 14:20 i2c-3
drwxr-xr-x 3 root root 0 Mar 10 14:20 i2c-4
drwxr-xr-x 3 root root 0 Mar 10 14:20 i2c-5
drwxr-xr-x 3 root root 0 Mar 10 14:20 i2c-6
lrwxrwxrwx 1 root root 0 Mar 10 14:09 iommu -> ../../virtual/iommu/dmar0
lrwxrwxrwx 1 root root 0 Mar 10 13:59 iommu_group -> ../../../kernel/iommu_groups/0
-r--r--r-- 1 root root 4096 Mar 10 14:09 irq
drwxr-xr-x 2 root root 0 Mar 10 13:58 link
-r--r--r-- 1 root root 4096 Mar 10 14:09 local_cpulist
-r--r--r-- 1 root root 4096 Mar 10 14:09 local_cpus
-r--r--r-- 1 root root 4096 Mar 10 13:58 modalias
-rw-r--r-- 1 root root 4096 Mar 10 14:09 msi_bus
drwxr-xr-x 2 root root 0 Mar 10 14:20 msi_irqs
-rw-r--r-- 1 root root 4096 Mar 10 13:59 numa_node
drwxr-xr-x 2 root root 0 Mar 10 13:58 power
-r--r--r-- 1 root root 4096 Mar 10 14:09 power_state
--w--w---- 1 root root 4096 Mar 10 14:09 remove
--w------- 1 root root 4096 Mar 10 14:09 rescan
--w------- 1 root root 4096 Mar 10 14:09 reset
-rw-r--r-- 1 root root 4096 Mar 10 14:09 reset_method
-r--r--r-- 1 root root 4096 Mar 10 13:59 resource
-rw------- 1 root root 4194304 Mar 10 14:09 resource0
-rw------- 1 root root 268435456 Mar 10 14:09 resource2
-rw------- 1 root root 268435456 Mar 10 14:09 resource2_wc
-rw------- 1 root root 64 Mar 10 14:09 resource4
-r--r--r-- 1 root root 4096 Mar 10 13:59 revision
-rw------- 1 root root 131072 Mar 10 14:11 rom
lrwxrwxrwx 1 root root 0 Mar 10 13:58 subsystem -> ../../../bus/pci
-r--r--r-- 1 root root 4096 Mar 10 13:59 subsystem_device
-r--r--r-- 1 root root 4096 Mar 10 13:59 subsystem_vendor
-rw-r--r-- 1 root root 4096 Mar 10 13:58 uevent
-r--r--r-- 1 root root 4096 Mar 10 13:58 vendor
The potential cause of ALL my final problems probably stem from having the wrong vBIOS ROM. At first, I attempted to get a suitable vBIOS ROM with the following:
# echo 1 > /sys/bus/pci/devices/0000:00:02.0/rom
# cat /sys/bus/pci/devices/0000:00:02.0/rom > rom
cat: '/sys/bus/pci/devices/0000:00:02.0/rom': Input/output error
# echo 0 > /sys/bus/pci/devices/0000:00:02.0/rom
# cat /sys/bus/pci/devices/0000:00:02.0/rom > rom
cat: '/sys/bus/pci/devices/0000:00:02.0/rom': Invalid argument
However, this approach had no joy (does anyone know the reason why)? I did attempt to research methods specifically using the bit-nibbler dd on a memory region BUT was not able to find anything specific to the i915.
Upon failing to get a suitable vBIOS ROM from the logical device tree, the following was cloned and compiled accordingly:
https://github.com/patmagauran/i915ovmfPkg
The files of relevance include the following (note that it appears that most of these files are 32-bit even though in the build directory it indicates 64-bit, which is confusing):
bdsmSize.bin: data
i915ovmf.debug: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped
i915ovmf.efi: PE32+ executable (EFI boot service driver) x86-64, for MS Windows, 3 sections
i915ovmf.rom: BIOS (ia32) ROM Ext. (111*512) instruction 0x00f10e00; at 0x1c PCI Intel device=0x1926 PRIOR, ProgIF=3, revision 3, code revision 0x3, code type 0x3, last ROM
i915vbios.rom: BIOS (ia32) ROM Ext. (39*512) instruction 0x00f10e00; at 0x1c PCI UNKNOWN (0x1f96) device=0x0778 PRIOR, revision 3, code revision 0x3, code type 0x3, last ROM
opregion.bin: data
All of these i915 related files (have interchangeably been used) are referenced in the QEMU/virt-manager XML configuration file as follows:
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</source>
<rom bar='on' file='/vm/macOS/i915/i915vbios.rom'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</source>
<rom bar='on'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
</hostdev>
...
<qemu:commandline>
<qemu:arg value='-fw_cfg'/>
<qemu:arg value='name=opt/igd-opregion,file=/vm/macOS/i915/opregion.bin'/>
<qemu:arg value='-fw_cfg'/>
<qemu:arg value='name=opt/igd-bdsm-size,file=/vm/macOS/i915/bdsmSize.bin'/>
<qemu:arg value='-device'/>
<qemu:arg value='isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc'/>
<qemu:arg value='-smbios'/>
<qemu:arg value='type=2'/>
<qemu:arg value='-usb'/>
<qemu:arg value='-device'/>
<qemu:arg value='usb-tablet'/>
<qemu:arg value='-device'/>
<qemu:arg value='usb-kbd'/>
<qemu:arg value='-cpu'/>
<qemu:arg value='Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check'/>
</qemu:commandline>
Does anybody have an idea of what may be causing me the problem with the blank screen from what seems like a well-tested comprehensive configuration? Should anyone be up for the challenge of resolving this issue and requires any further information please do let me know ASAP and I will provide what is required to further troubleshoot and resolve this issue accordingly. Many thanks in advance to anyone that helps in getting a resolution. -STace
-- FURTHER UPDATE TO ORIGINAL POST -------
Upon investigating further there is now a further update accordingly.
I recompiled the i915 vBIOS ROM source code but enforced 64-bit with the build environment source config containing:
export CFLAGS=“-m64”
export CXXFLAGS=“-m64”
export LDFLAGS=“-m64”
export CPPFLAGS=“-m64”
export CC=“gcc -m64”
export CXX=“g++ -m64”
However, this did not change the file type on the output binaries from the build! Worth investigating nonetheless in order to rule that out being the cause of my problems. Upon going through the website for the i915 vBIOS ROM there was a minor detail that was overlooked, that being the fact that according to…
https://github.com/patmagauran/i915ovmfPkg/wiki/Having-Problems%3F
The first bullet point states:
The iGPU MUST be at address 00:02.00 in the GUEST OS.
I therefore amended the relevant entries with the following changes:
<hostdev mode='subsystem' type='pci' managed=‘yes'>
<driver name=“vfio”/>
<source>
<address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</source>
<rom bar='on' file='/vm/macOS/i915/i915vbios.rom'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x2' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed=‘yes'>
<driver name=“vfio”/>
<source>
<address domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</source>
<rom bar='on'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</hostdev>
Notice the addition of the driver tag in the XML.
This allowed for direct address mirroring from the underlying source hardware to the virtual mappings accordingly. However, upon starting the VM from virt-manager the following error was discovered at the end of the log file:
2023-03-11T01:18:04.390922Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:00:02.0","id":"hostdev0","bus":"pcie.0","multifunction":true,"addr":"0x2","rombar":1,"romfile":"/vm/macOS/i915/i915ovmf.efi"}: IGD device 0000:00:02.0 cannot support legacy mode due to existing devices at address 1f.0
Initially, it was assumed that 1f.0 was a virtual address mapping but that is NOT the case. In fact, it references the following on the underlying hardware:
00:1f.0 ISA bridge: Intel Corporation 8 Series LPC Controller (rev 04)
So, the next logical progression was to research in further depth the error:
IGD device 0000:00:02.0 cannot support legacy mode due to existing devices at address 1f.0
This is the QEMU source code that generates the error in question:
+ /*
+ * We need to create an LPC/ISA bridge at PCI bus address 00:1f.0 that we
+ * can stuff host values into, so if there's already one there and it's not
+ * one we can hack on, legacy mode is no-go. Sorry Q35.
+ */
+ lpc_bridge = pci_find_device(pci_device_root_bus(&vdev->pdev),
+ 0, PCI_DEVFN(0x1f, 0));
+ if (lpc_bridge && !object_dynamic_cast(OBJECT(lpc_bridge),
+ "vfio-pci-igd-lpc-bridge")) {
+ error_report("IGD device %s cannot support legacy mode due to existing "
+ "devices at address 1f.0", vdev->vbasedev.name);
+ return;
}
