0

I'm currently setting up a CUDA-Workstation that is running on Debian GNU/Linux.

The system is equipped with two nVidia GFX cards (for the CUDA stuff), and features an onboard VGA adapator.

In the end, the machine is going to live in a remote location and will be managed via IPMI, which means that I would like to have the onboard VGA as my primary display device (while the nVidia cards do feature a total of 8 DP/HDMI outputs, they are not going to be connected; the onboard VGA is not going to be connected either, but is captured by IPMI/BMC so I can access it via KVM-over-IP).

So what I need is the Linux framebuffer console to show up on the VGA.

Currently I'm facing two problems here:

onboard VGA only used in early boot

After setting the primary output device in the BIOS to "onboard" (from "dedicated gfx card"), the onboard VGA works nicely in the early boot phase: I can see POST, access the BIOS, handle the UEFI boot loader (grub2 in my case) and start the OS. Hooray!

After that however, the output freezes and I am left with the grub2-printout (and nothing else):

   Booting Debian GNU/Linux

Loading Linux 6.1.0-5-amd64 ...
Loading initial ramdisk ...

And it stays like that until the system goes down.

This is only an output issue, the system boots up just fine (and I can ssh into the machine and do whatever). I can also use the attached USB keyboard to log in via the login on tty1 and issue commands there... but the screen stays frozen (so I don't see what I type and what the system answers)

I've tried other kernels (5.x, 6.x) and distros (like grml, which uses isolinux instead of grub), but the problem persists:

  • i can view the bootloader
  • but as soon as the actual system starts, the output freezes

I've tried adding noplymouth, textonly, text and vga=ask to the kernel-cmdline, but to no avail...

dedicated GFX-card not showing boot messages with newer kernels

If I switch to using one of my dedicated GFX cards as the primary device (in the BIOS), things are slightly different.

With Debian/bullseye (current stable Debian release), using the dedicated GFX output works just fine. However, if I upgrade the kernel to 6.x (either via Debian/bullseye-backports; or by upgrading to Debian/bookworm (the upcoming stable release), I no longer see any boot messages or login screen.

Instead my screen only shows this:

[    2.418707] xhci_hcd 0000:23:00.0: init 0000:23:00.0 fail, -16
[    2.418805] xhci_hcd 0000:29:00.1: init 0000:29:00.1 fail, -16
[    2.418861] xhci_hcd 0000:29:00.3: init 0000:29:00.3 fail, -16
/dev/nvme0n1p2: clean, 78970/122003456 files, 8917355/487997184 blocks

Also, I cannot switch to other terminals ((Ctrl+)Alt+F2). But again, keyboard input actually works (only the display is frozen).

I can switch between the working console and a non-working by booting into a linux-5.x kernel (working) resp. a linux-6.x kernel (broken).

While my system is equipped with 2 nVidia GFX cards, there are currently no dedicated (proprietary) drivers installed. So I see this behaviour with a pristine Debian main installation.

However, in the end I will have to use Debian/bookworm (with the linux-6.x kernel), as i need the newest nVidia drivers to make use of the CUDA cards. (and before you ask: the problem persists if I install the proprietary drivers).

I cannot reproduce the problem on my Laptop (which runs Debian/sid - thus has a kernel 6.1.0-5-amd64; all tty work as expected) or on a freshly installed VM.

NOTE: There are other drawbacks to using the dedicated GFX card anyhow: I won't be able to access its output via IPMI/BMC once the machine is in the remote location. So this part of the problem is somewhat academic...

system setup

component name
motherboard ASUS Pro WS WRX80E-SAGE SE WIFI
CPU AMD Ryzen Threadripper PRO 5965WX
memory 132GB
onboard VGA ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
dedicated GFX 2* NVIDIA Corporation AD102 [GeForce RTX 4090] (rev a1)
OS Debian bullseye (fresh installation), resp bookworm (apt dist-upgrade) from bullseye

I conducted the tests right after doing a fresh minimal installation of Debian/bullseye, and then upgraded to Debian/bookworm.

No X-server or similar is installed. No proprietary drivers are installed.

diagnostic output

after installing the proprietary nvidia drivers (which I did by now), here's some relevant system output:

# uname -a
Linux petrou 6.1.0-5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.12-1 (2023-02-15) x86_64 GNU/Linux
# dmesg | egrep -i "(vga|console|efifb)"
[    0.078684] Console: colour dummy device 80x25
[    0.078873] printk: console [tty0] enabled
[    1.888692] pci 0000:28:00.0: BAR 0: assigned to efifb
[    1.903739] pci 0000:41:00.0: vgaarb: setting as boot VGA device
[    1.903739] pci 0000:41:00.0: vgaarb: bridge control possible
[    1.903739] pci 0000:41:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    1.903739] pci 0000:28:00.0: vgaarb: setting as boot VGA device (overriding previous)
[    1.903739] pci 0000:28:00.0: vgaarb: bridge control possible
[    1.903739] pci 0000:28:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    1.903739] pci 0000:2b:00.0: vgaarb: bridge control possible
[    1.903739] pci 0000:2b:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    1.903739] vgaarb: loaded
[    2.144244] efifb: probing for efifb
[    2.144245] pci 0000:28:00.0: BAR has moved, updating efifb address
[    2.144247] efifb: cannot reserve video memory at 0x0
[    2.144249] efifb: video memory @ 0x0 spans multiple EFI memory regions
[    5.273136] systemd[1]: Started systemd-ask-password-console.path - Dispatch Password Requests to Console Directory Watch.
[    5.277559] systemd[1]: Starting keyboard-setup.service - Set the console keyboard layout...
[    5.315244] systemd[1]: Finished keyboard-setup.service - Set the console keyboard layout.
[    5.521188] snd_hda_intel 0000:41:00.1: Handle vga_switcheroo audio client
[    5.521370] snd_hda_intel 0000:2b:00.1: Handle vga_switcheroo audio client
[    5.592959] nvidia 0000:41:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.639620] nvidia 0000:2b:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none

# lsmod | egrep -i "(vga|fb)"

# cat /proc/fb

# cat /etc/modprobe.d/blacklist-framebuffer.conf
cat: /etc/modprobe.d/blacklist-framebuffer.conf: No such file or directory

# find /etc/modprobe.d/ -not -type d -exec grep blacklist {} +
/etc/modprobe.d/nvidia-blacklists-nouveau.conf:blacklist nouveau

# lspci  | grep -i vga
28:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
2b:00.0 VGA compatible controller: NVIDIA Corporation AD102 [GeForce RTX 4090] (rev a1)
41:00.0 VGA compatible controller: NVIDIA Corporation AD102 [GeForce RTX 4090] (rev a1)

# lspci -vvnn -s 28:00.0
28:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41) (prog-if 00 [VGA controller])
    Subsystem: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000]
    Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 75
    IOMMU group: 48
    Region 2: I/O ports at 1000 [size=128]
    Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Kernel modules: ast

So what is the question?

Well: How do I get a full console output for my terminal on my onboard VGA (or at least on the dedicated GFX card?)

Improve this question
5
  • less /etc/modprobe.d/blacklist-framebuffer.conf ?
    – Hannu
    Mar 1 at 20:12
  • "I've tried adding noplymouth,textonly, text and vga=ask to the kernel-cmdline" -- Huh? Where are the first three documented as valid Linux kernel parameters? Did you try console=tty0? Have you bothered to thoroughly review the boot log for "VGA" and "console" status messages (e.g. using the dmesg command)? Beside "console output", are you also expecting a login console, which is a completely different specification?
    – sawdust
    Mar 2 at 1:42
  • @Hannu no such file exists (and i've added this info in the "diagnostic output" section)
    – umläute
    Mar 2 at 11:49
  • @sawdust the boot options are from "random sources on the internet" (just to see how desperate I am...); I did bother a lot with dmesg (and have now added that info to the "diagnostic output" section). i have not tried setting console=tty0 explicitely (but I think the dmesg output hints that this is already the case anyhow).
    – umläute
    Mar 2 at 11:51
  • @sawdust: the "login console" is secondary (as long as booting into emergency mode would give me a usable console). AFAICT, there is a "login console", i just don't see it (with the non-functional VGA output)
    – umläute
    Mar 2 at 11:54

1 Answer 1

Reset to default
2

Finally I found a solution for my problem.

TL;DR

adding pci=nommconf to the kernel commandline solved my problems.

longer version

It turns out, that there was some issue with "Memory-Mapped PCI Configuration Space", which somehow played havoc with both the onboard VGA card, and the onboard SATA-controller (which i thought was unrelated, so I didn't mention it in my Q; in short: I have two disks, an NVMe/M.2 disk which works nicely, and an additional SATA disk, which did not show up at all).

The SATA disk not showing up (at least under Linux) seems to be a known problem. While this forum-post already mentions the pci=nommconf, I either missed it or it I did something wrong when applying it (or i just gave up with that forum-thread after unsuccessfully trying to downgrade my BIOS, resulting in a machine that wouldn't even POST anymore).

In any case, I found yet another forum post, which mentions that they had to add the following three kernel-parameters to get their nvidia GPU passed through to VMs (which I'm not interested in, as I need the nVidia cards on the host anyhow):

  • amd-iommu=on
  • iommu=pt
  • pci=nommconf (optional, for their problem)

Adding all three parameters, I finally got all the boot-messages on the VGA-output (and on the IPMI/BMC!, even though this one is slightly overhelmed by the verbosity of no-`quiet´; but it helps resetting the KVM-over-IP).

Adding the parameters also made my SATA-disk magically appear (and I could cancel my ordering of a 2TB M.2 NVMe disk :-))

It turns out that I don't really need the first two arguments, as amd-iommu=on is the default anyhow (at least on stock Debian kernels), and I don't need the passthrough (iommu=pt) in my use-case.

afaict, the relevant dmesg lines (indicating that something was seriously wrong with the memory-mapping of the efi-framebuffer) were these:

[    2.144244] efifb: probing for efifb
[    2.144245] pci 0000:28:00.0: BAR has moved, updating efifb address
[    2.144247] efifb: cannot reserve video memory at 0x0
[    2.144249] efifb: video memory @ 0x0 spans multiple EFI memory regions
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .