15

I've been dealing with a problem where if I don't enable "Above 4G decoding" in the BIOS settings, my server system won't boot up with a certain number of storage devices.

Google results give this: "The definition of “Above 4G decoding” is to allow the user to enable or disable memory mapped I/O for a 64-bit PCIe device to 4GB or greater address space, because the primary VGA card should always be mapped below 4GB address." (https://www.asus.com/support/FAQ/1004170/)

But what does this mean, for someone who doesn't know much about computers?

2
  • 3
    It means that the system memory assigned to your VGA card must below the 4 GB address space unless this option is enabled. Depending on the exact specifications of your server, this might not be possible, if the option is disabled. You should leave it enabled based on the fact your system will not boot with it disabled.
    – Ramhound
    Aug 9, 2017 at 17:09
  • 1
    @Ramhound You should make this an answer, not a comment. Thanks though!
    – jmoon
    Aug 9, 2017 at 18:10

5 Answers 5

15

Since there was never an official answer and this is one of the first Google results:

This allows 64-bit PCIe devices to use addresses in the 64-bit address space. Since 32-bit operating systems cannot access the 64-bit address space*, this option is for compatibility reasons. In most cases, it does not need to be enabled. In my experience, most motherboards default to having this setting off. There's generally no harm in enabling it if you have a 64-bit OS.

Generally, people enable it when they have many PCIe cards installed, such as with triple and quad SLI and using GPGPU compute modules such as NVIDIA's Tesla. It's more commonly used in workstations and servers.

Since neither setting should cause issues on a 64-bit operating system, you can enable or disable it without any ill effect most of the time.
______________
* Note that a 32-bit processor, or a processor in 32-bit mode, can access 232 bytes, which is 22×230 = 4 gibibytes (4 GiB).

4
  • 2
    Likely the OP has some unusual PCIe devices with large amounts of onboard high-speed memory that the BIOS thinks the CPU needs access to. The BIOS might be wrong about that, in which case a BIOS update might fix it. But the BIOS might also be right. I had this same issue with a desktop with a Xeon Phi coprocessor with lots of onboard RAM. In that case, the BIOS was right. Jan 6, 2019 at 22:48
  • 1
    Interesting point: In my machine's firmware settings this is under the "boot options". If that actually means anything, then an OS should have no trouble accessing PCIe devices that are at addresses above 4 GiB, even if this option is turned off. You should only need this turned on if you need your firmware's boot code to see those devices. Mar 14, 2019 at 1:58
  • The BIOS will almost certainly not share the device memory. The device itself needs to be addressable but will behave like a server. The issue is that the OS needs to boot, and during the boot it may need to address the primary GPU in a 32 bit address space. But the more cards you add the less of that space there is. Miners need to enable this because they have lots of cards (duh!) if each takes 700K it adds up.
    – mckenzm
    Apr 5, 2021 at 2:18
  • Actually: this setting seems to be necessary for Resizable BAR / SAM and "Above 4G Decoding" causes a libvirt+qemu Windows 10 VM (q35) to not boot :(, which is why I have to disable it. At the moment it's not a problem, but it might be if it's not resolved somehow.
    – Igor
    May 10, 2021 at 0:45
5

On x86 PCs, the memory space is shared in the system. System memory maps from the bottom up (0 -> xGB) & PCI memory space is in the upper part of system memory space (xxx --> FFFFFFFF 4GB-1). Note that "memory space" is not "memory". It's just address ranges for HW to decode & claim transactions.

This is usually ok because PCI/PCIe devices generally request a reasonable amount of memory for their PCI BAR (Base Address Range), such as 64K or less. BARs are where things like device-specific registers reside and what a device driver would access to control the device. Very large BARs don't apply to most devices because they use DMA engines to transfer data instead of the host CPU performing PIO accesses (very poor performance), so they don't need to map their internal storage to BARs.

Once you start using GPU type devices, for example, the higher end ones request very large PCI BAR spaces, such as 32GB or more. If your PCI memory space is limited to 32-bit, then there's no available address range to fit 32GB in. So, BIOS PCI enumeration started to support 64-bit PCI BAR assignment during enumeration. This generally started about 12 years ago and some of the first algorithms to support it still had bugs or limitations. Some systems would hang if a BAR was larger than 8GB. PCI enumeration algorithms have improved since then.

So, the "Above 4GB decoding" means that the BIOS PCI enumeration is "allowed" to assign PCI BARs memory ranges above 4GB (32-bit max). It may even do that for small PCI BARs, as long as they report themselves as 64-bit.

Note that in PCI/PCIe devices, PCI BARs are 32-bit. If a PCI BAR wants to support 64-bit, it "combines" 2 32-bit BARs to store the 64-bit address. A device with "64-bit PCI BAR 2" means that its BAR 2 is low 32b of the assigned address & BAR 3 then becomes the upper 32b.

Also note that 32-bit x86 can still access PCI BARs above 4GB due to PAE.

2

If you're running a 32 bit operating system, you will almost certainly need to turn this OFF, otherwise some of your hardware may not be seen at all by the OS. Since this will map your video memory into the ~3.5Gb available, you should use whatever tools came with it, and make sure it's not using a lot more memory than you need, since that memory won't be available to your operating system and apps. Most can be restricted to 128 or 256 Kb. If you're using a 64 bit OS, with 8Gb or more, you can probably leave it off, unless you're gaming with an expensive video card.

1

This option allows Tesla cards to map their video memory directly into the memory range of the machine where it's plugged into.

Since a lot of memory on a card won't fit below the 4GB range ( eg: 12/24GB vRAM on Teslas or newer Quadro/GF models ) the BIOS must allow the range declaration of the card above the 32Bit line ( being 4GB )

This is mandatory for Tesla and MultiGPU systems that map more and/or above 4GB.

2
  • Beware of such logic. The VRAM is no more relevant to the BIOS address space than your hard disk size. The GPU may as well be a daughter computer. It may however have I2C and USB onboard, and these need to be mapped. It is mandatory where there is more than 3 GPU's, and recommended for 2. It is the number of GPU's. not their RAM.
    – mckenzm
    Apr 5, 2021 at 5:19
  • photos.app.goo.gl/DZyv3YbqqRiKruWs7 Above 4G in action on an Ubuntu machine - 16 GB Mapping zone for one GPU ( the K80 has 2 ) Oct 31, 2021 at 11:55
1

Apparently Above 4G decoding is necessary to unlock "Smart Access Memory" with AMD's latest NAVI GPU... which allows for upto 16% increase in performance in games.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.