QEMU Radeon GPU passthrough with error code 43 after install driver

Question

I'm trying to do a single GPU passthrough and I actually got some results, it worked before having its driver installed, but after that it only shows the boot loading and then it turns into a black screen. Using VNC I managed to get this image:

I'm running it with:

qemu-system-x86_64 \
    -name 'Win10' \
    -machine type=q35,accel=kvm,kernel_irqchip=on,vmport=off \
    -m 32G \
    -mem-path /run/hugepages/kvm \
    -cpu host,kvm=off,+kvm_pv_unhalt,+kvm_pv_eoi,topoext,amd-stibp=off,hv_relaxed,hv-vapic,hv-spinlocks=8191,hv_vpindex,hv-runtime,hv_time,hv-synic,hv-stimer,hv-vendor-id=123456789ab,hv-reset,hv-frequencies \
    -smp 16,sockets=1,cores=8,threads=2 \
    -nodefaults \
    -bios /usr/share/ovmf/OVMF.fd \
    -smbios 'type=1,uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
    -netdev user,id=n0,hostfwd=tcp::5900-:5900 \
    -device e1000,netdev=n0 \
    -object input-linux,id=kbd0,evdev=/dev/input/by-id/usb-SONiX_USB_DEVICE-event-kbd,grab_all=on,repeat=on \
    -object input-linux,id=mouse0,evdev=/dev/input/by-id/usb-Razer_Razer_DeathAdder_Essential-event-mouse \
    -device virtio-mouse-pci \
    -device virtio-keyboard-pci \
    -monitor telnet::5555,server,nowait \
    -nographic \
    -vga none \
    -display none \
    -device vfio-pci,host=2d:00.0,multifunction=on,x-vga=on,x-pci-sub-vendor-id=0x1458,x-pci-sub-device-id=0x232c \
    -device vfio-pci,host=2d:00.1 \
    -rtc base=localtime,clock=host,driftfix=slew \
    -object iothread,id=io0 \
    -drive 'if=none,id=disk0,format=qcow2,aio=threads,cache=writeback,l2-cache-size=64M,cache-clean-interval=900,file=win.qcow2' \
    -device virtio-blk-pci,drive=disk0,iothread=io0 \
    -drive "media=cdrom,file=$HOME/virtio.iso" \
    -boot c

• kernel_irqchip=on was suggested here and here
• vmport=off was suggested here
• kvm=off and hv-vendor-id= were suggested here and here
• +kvm_pv_unhalt and +kvm_pv_eoi were suggested here
• topoext was suggested here
• amd-stibp=off was suggested here
• hv_relaxed, hv-vapic, hv-spinlocks, hv_vpindex, hv-runtime, hv_time, hv-synic, hv-stimer, hv-reset, and hv-frequencies were suggested here, here and here
• x-pci-sub-vendor-id and x-pci-sub-device-id were suggested here
• I also tried romfile= in the device extracting the rom with GPU-Z as suggested here, here and here.

I also have my kernel command appended with amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb video=vesafb:off,efifb:off". This was suggested here, here and here.

Before running the VM I run the following script:

sysctl vm.nr_hugepages=16384 vm.min_free_kbytes=112640 kernel.shmmax=34359738368

sync
echo 3 > /proc/sys/vm/drop_caches
echo 1 > /proc/sys/vm/compact_memory

echo 0 > /sys/module/kvm/parameters/report_ignored_msrs
echo 1 > /sys/module/kvm/parameters/ignore_msrs
echo 1 > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts


systemctl stop lightdm
systemctl isolate multi-user

while systemctl is-active --quiet lightdm; do
    sleep 1
done

echo 0 > /sys/class/vtconsole/vtcon1/bind

echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind || :
echo 1 > /sys/bus/platform/drivers/efi-framebuffer/unbind || :

echo '0000:2d:00.1' > '/sys/bus/pci/devices/0000:2d:00.1/driver/unbind' || :

echo 'vfio-pci' > '/sys/bus/pci/devices/0000:2d:00.0/driver_override'
echo 'vfio-pci' > '/sys/bus/pci/devices/0000:2d:00.1/driver_override'
echo '1002 ab28' > /sys/bus/pci/drivers/vfio-pci/new_id
echo '1002 73bf' > /sys/bus/pci/drivers/vfio-pci/new_id

• it was suggested mostly here.
• vm.nr_hugepages was suggested here
• drop_caches and compact_memory were suggested here
• ignore_msrs and report_ignored_msrs were suggested here and here.
• allow_unsafe_interrupts was suggested here and here

As you can see, I've followed several recommendations, even for those unrelated to code 43, I'm running out of ideas how to solve that...

PC Specifications

• OS: Ubuntu 20.04.4 Kernel 5.15.0-41-generic
• CPU: AMD Ryzen 9 5900X
• GPU: AMD Radeon RX 6900 XT
• Motherboard: MSI B550 Gaming Carbon WiFi
• QEMU: 4.2.1 (I've also tried with 7.0.0)

Answer 1

I was able to resolve the issue with my RX 6900 XT by disabling Resizable BAR in the BIOS.

Answer 2

take a try with bellow grub command

amd_iommu=on iommu=pt nofb nomodeset video=vesafb:off,efifb:off vga=normal i915.modeset=0

or

amd_iommu=on iommu=pt nofb nomodeset video=efifb:off vga=normal i915.modeset=0