Can some programs really access “unavailable” RAM on a 32-bit system?

Question

Question

I’ve seen some programs (especially RAM-drive software) that purport to being able to access the “unavailable RAM” on 32-bit systems (including XP). I’m wondering if they actually can, and if so, how they manage to do it.

Technical

There are several different way in which the amount of RAM installed in a 32-bit Windows system can be limited (simplified here):

• 2:2 (2GB) Virtual Address-space

  Because of the way that Windows partitions the address space, programs can only allocate a block of memory up to 2GB in size. The /3GB switch can mitigate this to some degree by changing the ratio from 2:2 to 3:1.

• 3.xGB - 4GB

  Due to hardware address mapping (MMIO), some of the address space will be used up by other devices installed (or built-into) the system. Memory remapping may help with this on systems that support it.

• 4GB+

  Because a 32-bit pointer can only point to up to 2³² addresses, it can only point to up to 4GB worth of data. Using PAE mode can help with this by using larger pointers and “pages” (much like the old 16-bit DOS segment:offset memory scheme, which is ironic since protected mode was supposed to be better than real mode).

• Microsoft’s Marketing Limits

  In addition to mathematical limits of hardware and software, Microsoft also imposes an artificial limit on Windows (*including 64-bit versions) so that they can market different editions for different purposes—and prices.

Specifics and Examples

My question is not about how Windows or regular programs (like Notepad) are affected by any of this. I’m already familiar with these limitations and their typical work-arounds (including the Russian patch that makes the Windows 7 kernel more like the Windows Server 2003 kernel), and I am not interested in rehashing them.

My question is about programs that claim to be able to overcome one or more of these limits.

I think I’ve seen other programs make claims like this, but I’m certain that some RAM-drive software does. Some examples include Raxco, DATARam, and Gavotte. I’ve previously tried a couple of RAM-drive programs that made such claims (I think Gavotte was one of them) but was not able to get any to actually access the extra memory.

Claims

Raxco makes the following claim:

Not only can RamDisk Plus create RAM disks with the hidden RAM above 4GB; it can also use the inaccessible memory between 3.2GB and 4GB.

And later on they mention that:

Keep in mind, this can only be done if Windows reports the hidden RAM above 4GB.

The first statement is good because unlike most of these programs, it is not vague about which limit(s) it overcomes.

However, like most of these programs, it is vague about what is required (hardware- and software-wise) for it to access the extra memory, and says nothing about how it works or what kind of side-effects it has (for example, if it merely adds the /3GB switch or enables PAE mode, then it is not “overcoming” the problem, it is enabling a special Windows mode—that would affect the OS and all programs—and accommodating that).

Jens’ discussion of the Gavotte RAM-drive seems to indicate that they (Gavotte at least) do in fact work by using PAE. It also mentions the article Geisterspeicher from the German magazine c't as the source of the breakthrough. Unfortunately not only is the article not free, but my German is not yet fluent enough to read a technical article, so I don’t know if it gives details of how it works or its requirements.

Summary

Does anybody know if there is any truth to these programs’ claims? Has anybody actually seen it work? If so, what is required to make them work? Also, how exactly do they pull it off (do they affect the whole system and all programs by enabling /3GB, PAE, etc. or are they doing something special like using extra-large pointers and accessing hardware directly?

Answer 1

I have seen some programs (especially RAM-drive software) that purport to being able to access the “unavailable RAM” on 32-bit systems. That is, even though Windows cannot see or access some of the 4GB that is installed on a 32-bit system, these programs claim that they can.

I guess technically this is possible if the programs manages to insert itself early enough to activate either PAE or 64 bit mode, and then emulate normal operation to the OS before it finishes loading.

On my 32-bit system with 4GB, Windows sees only 3.20GB, but so does the BIOS.

This is because a 32 bit system has 4 GB address space. Part of that (ideally 512 MB or less) should be used for the RAM. The rest is for PCI address space, virtual memory addresses etc.

It is not a matter of using PAE or the /3G switch, because it’s not a Windows limitation, it is a motherboard limitation. If the chipset and memory controller can’t access beyond that, then I don’t see how Windows or any software can either, even if they access the hardware directly.

True. If your memory controller or the motherboard is limited to 4 GB then that is that. Game over.

I know that using PAE requires using either a server or 64-bit edition of Windows

No. This is wrong.

PEA works on a 32 bit OS. It uses 36 bit addressing. Think of it as 16 (2⁴) windows of 4 GB (2³²) each. Or think of it as a book with 16 pages. You can not see more than one page at the same time, but you can turn the pages.

(though I don’t see how even these versions of Windows can access what the BIOS cannot). However these programs say nothing about that and imply (or outright say) that they work for normal users with consumer versions of Windows.

PAE is an option on these windows operating systems, assuming that the hardware supports it.

• Windows 7 (32 bit only)
• Windows Server 2008 (32-bit only)
• Windows Vista (32-bit only)
• Windows Server 2003 (32-bit only)
• Windows XP (32-bit only)

Source.

As you can see that is on 32 bit OS's only. Both the server and the consumer versions. I seem to recall that it was later removed from windows 7 SP1, though that is not mentioned on the MS site.

I have tried a couple of these programs (specifically RAM-drives), but was not able to have it access the upper memory. Does anybody know if there is any truth to these programs’ claims? Has anybody actually seen it work? And if so, how exactly do they pull it off?

Without knowing which programs we can not answer that.

However they might work on a motherboard which supports PAE, but not on a motherboard which either lacks the copper traces for it, or whose memory controller does not support it. (e.g. old CPU's, old chipsets, atom CPU's older than Pineview or Diamondville).

Answer 2

Ram-Disks using upper memory in 32bit systems.

It must be seen in the Bios, to do that on many boards you make sure that the board itself is switched to extended memory. The 32Bit OS, must be in PAE mode. Logically you would have more memory than 4Gig.

The reason why we get LESS than 4Gig in a 32 bit system is the total address space is also used by the video cards memory space to set out the video cards ram. The more GPU card ram there is laid out in that space , the less "virtual memory" address space for the system ram.

There is no limitations like that in 64bit system because the video cards 64bit driver could potentially allocate space outside of the 32bit address limitations, Not that It would need to, because the rest of the ram still can have ample virtual memory spaces alloted to it.

----Video ram---system ram----|32bit end|----extended--

Below is a C&P post from when I did extended ram-disk in 32bit system myself a long time ago, It was supposed to cover all the things that could go wrong. I have XP 32bit set up for a 4gig ram-disk, using the extended memory.

---

Use the Other 4Gig you can not access in a 32bit system, as a ramdisk.

For years I have had a ramdisk on and off in my system, toss internet temps on them. The web , which should be bottlenecked by the speed of the connection, goes a Lot faster. aparentally the pages are assembled on the faster disk Prior to showing it, make it faster.

Eventually I get boored with the ramdisk Taking up my limited 32bit spaces, or I am running photoshop with huge pictures, or some other thing, and I turn the ramdisk off. Eventually I removed it altogether , because it was not used enough.

While checking out some Windows 7 64bit stuff for Windows 7, NOT for my xp32bit. I read that these ramdisks can now be setup to USE the unusable 4 gig that is after the windows 32bit max allocations :-)

I am using the DataRam one, here are 2 simple tips from testing it. I set my Boot.ini to run in PAE mode (otherwise it cant use the ram beyond the system) I already had the system booting up in EXTENDED because of the 64bit system. (That is a option in the motherboards to be running in extended memory.) After that was all done, I was able to have the ramdisk snag the Last 4gig of my 8gig of memory.

More info: All your drivers and all have to be able to run in extended/Pae to get the OS into PAE mode. If you have not ever done that, you should probably test your system in PAE mode first. at first my system crashed because I supidly had the ramdisk driver itself already still set for the normal ram spaces. I jumped back in to the NOPAE mode, switched the driver off, then back into PAE mode to configure it under PAE mode.

This is what my Boot.ini looks like , so I can hop between the 2 modes when I screw up.

[boot loader]
timeout=3
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="PAE XP Professional" /fastdetect /pae
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="XP Professional No Pae" /fastdetect /noexecute=alwaysoff /NOPAE

Just so you have some idea or can search for correct ways to do that, making no assumptions that mine is correct for you. It just shows that I have a backup method for entering the OS.

It was the first time I had use of ALL my 8gig of memory in my old 32bit XP, soo I thought that was pretty dang cool. and I think that others would also. Especially when things like w7 64bit suck so bad for some things.

The ramdisk was "free" it puts up one add to buy memory when configuring, and I am sure that Other ramdisks can now do this. it was only free for the first 4gig, but quite sufficient for my needs.

The ramdisk is about 70 times faster than my normal sata drive, instead of 100, which an older ramdisk was, so there seems to be a tiny bit more overhead.

Answer 3

All of these programs require a system that supports PAE and a motherboard that remaps some physical memory above the 4GB mark. During early Windows bootup, these programs claim all the memory above the 4GB mark. Basically, if the memory would work with an operating system that fully supported PAE, then these programs can find it.

If the BIOS doesn't see the memory, it's not mapping it above the 4GB mark, so it's not going to work. High memory only exists if you have more than 4GB of physical RAM or your BIOS remaps above the 4GB mark the RAM that's shadowed by hardware mappings. If neither of these things happen, there is no high memory.

The majority of consumer motherboards that don't support any 64-bit CPUs don't support more than 4GB of physical RAM either, nor do they support remapping memory shadowed by hardware mappings due to chipset limitations. So there's a pretty limited set of cases where this works -- it's mostly a marketing gimmick, promising something for nothing.

Answer 4

When I re-wrote the question to clarify what I’m asking and to stop people from trying to answer a different question (though some of the others provided bits of the whole answer), I added a bunch of links to different pages that explain the various limitations and work-arounds. During this process, I happened to see a few different paragraphs in the right order with the right timing to be able to connect them, which when taken together, explain how the programs access the extra memory and what’s required for it work. I’ll list my findings here. The two most revealing parts are as follows:

Jens Scheffler says of Gavotte:

Depending on the used hardware and BIOS configuration the non-usable memory between 3GB and 4GB (this is a area for reserved addresses for physical devices) is remapped to the area above 4GB. Windows XP/Vista/7 32Bit editions are limited to 4GB RAM addresses so the memory above 4GB is just "unused".

The Gavotte RAMDisk is able to set the RAMDisk in the area above 4GB memory addresses and can enable the usage of this area for other purposes.

Microsoft explains in a knowledge base article why 32-bit Vista does not show 4GB and how it can be made to:

For Windows Vista to use all 4 GB of memory on a computer that has 4 GB of memory installed, the computer must meet the following requirements:

• The chipset must support at least 8 GB of address space. Chipsets that have this capability include the following:
  • Intel 975X
  • Intel P965
  • Intel 955X on Socket 775
  • Chipsets that support AMD processors that use socket F, socket 940, socket 939, or socket AM2. These chipsets include any AMD socket and CPU combination in which the memory controller resides in the CPU.

The CPU must support the x64 instruction set. The AMD64 CPU and the Intel EM64T CPU support this instruction set. The BIOS must support the memory remapping feature. The memory remapping feature allows for the segment of system memory that was previously overwritten by the Peripheral Component Interconnect (PCI) configuration space to be remapped above the 4 GB address line. This feature must be enabled in the BIOS configuration utility on the computer. View your computer product documentation for instructions that explain how to enable this feature. Many consumer-oriented computers may not support the memory remapping feature. No standard terminology is used in documentation or in BIOS configuration utilities for this feature. Therefore, you may have to read the descriptions of the various BIOS configuration settings that are available to determine whether any of the settings enable the memory remapping feature. An x64 (64-bit) version of Windows Vista must be used. Contact the computer vendor to determine whether your computer meets these requirements.

To summarize how it works:

• The BIOS maps conflicting memory addresses (caused by MMIO) to addresses above 4GB
• Windows runs in PAE mode to support large pointers in software
• The program uses large pointers to access “high” memory (either between 3-4GB or above 4GB)

There are two requirements for this to work:

• The hardware (BIOS) must support memory-remapping. Look in the BIOS for settings labeled remap or memory hole.

• Windows must be set to PAE mode which means that all programs will be affected.

There are some important factors and issues to note:

• If your system does not support memory-remapping, then the programs will not be able to access the extra memory and everything (including Linux) will be limited to 3.xGB of RAM because that is all that is exposed at the hardware level. See the list mentioned in the KB article for the minimum chipsets that support this (mine is 865, so I’m out of luck; boo! that explains why the programs couldn’t access my extra RAM).

  • It’s not uncommon for a 32-bit system to not support memory-remapping because by the time that adding 4GB became affordable to the masses, 64-bit systems were already starting to become standard, so not many motherboard manufacturers bothered to implement it (at least not in their low- to mid-range models).

• Using PAE mode might affect system stability, especially with some drivers.

  • Due to driver compatibility issues, Microsoft specifically removed the ability to access memory beyond 4GB in XP SP2 (Address Windowing Extensions are retained). This casts serious doubt on the ones that claim to be able to do this on XP, specifically the ones that throw in SP2, SP3, and the kitchen sink as an absent-minded matter of meaningless coverage (read keyword spamming) rather than actual, tested platforms.

• The Microsoft KB article gives this unpleasant clarification:

  Note When the physical RAM that is installed on a computer equals the address space that is supported by the chipset, the total system memory that is available to the operating system is always less than the physical RAM that is installed. For example, consider a computer that has an Intel 975X chipset that supports 8 GB of address space. If you install 8 GB of RAM, the system memory that is available to the operating system will be reduced by the PCI configuration requirements. In this scenario, PCI configuration requirements reduce the memory that is available to the operating system by an amount that is between approximately 200 MB and approximately 1 GB. The reduction depends on the configuration.

  This means that you can never really “max out” a 32-bit system. The closest that you can come is by putting 4GB in a motherboard that supports memory-remapping and using a RAM-drive program that can make use of the excess.

Well computers have certainly gotten a lot more complex in the past decade or so and a lot of the complexities are unfortunately intricate, intertwined, and use confusing terminology that mixes things up. This one issue however seems to be resolved. The answer to whether a program can actually use the excess memory in a 32-bit system is yes, if you have the right hardware and can put Windows into PAE mode (so no exotic hardware with unusual or obsolete drivers). Otherwise, the answer (like for me) is no, they can’t.

Answer 5

Umm..basically I'm running Windows XP, with a Gavotte Ramdrive.. 3.2 GB free memory under Windows XP (none of it used by Ramdrive, basically max for Windows XP) Then I got a Ramdrive of 12GB, on which pagefile is located (4GB) I used to have an APU-5800K, with Gigabyte F2A55-M mainboard.. Now it's a diff mainboard, still using same setup, but with extra videocard. The OS does/did not have /PAE added to it. 16 GB Total system memory. It runs really well, specially older software that does/can not use more than a single core. The harddrive always being bottleneck on those, installing said software Like X3-The Threat or other such, greatly improves load times. (Going from sector to sector, and other long loads) Basically the following applies: IDE < SATA < SSD < Ramdrive. It really works that much faster than even SSD's. And since Virtual Memory is also used from it, any that is ever used loads at Ram speeds. I do not overclock (bad for components/processor shortens lifetime) The ram I use is 1866, also very usefull for the onboard GPU (HD7660D) and now with the graphics card, I reduced the load on <4GB to 0. So 3.2GB (maxed) under windows XP, super fast load times for any gamingz.. :D Also, no older softare is able to use >4GB..so yeah..it works like dream.. Instead of having 30 loads on extreme large builds in stuff like CNC3, X-series, the older but still bigger shooters, Fallouts and more..

• ish happeh..

I hope this helps..

Note: I did have to tweak some stuff on my mainboard's BIOS settings, in order to make it work, but once it does..BANG !..there goes soundbarrier..

Hmm...I don't wanna spend time doing bullet point formatting this..sowwies..

Sowwies... I forgot to mention the actual answer.. YES, it's possible to use more than 4Gb when you are on 32 Bit windows. Gavotte Ramdrive..guud.. BTW. when installing that software, don't change naything in the Ramdrive setup utility, just run once. It auto loactes any unused memory and then adds it to the Ramdrive..Just set to Ramdrive first.. (that's how it works for me anyway..) Good luck and Ramdrive speed..

AMD APU-A5800K

16GB physical memory (1866 for optimal APU performance) (You can also use the 21xx ones, but I never tried that, for even faster internal GPU performance, as well as still faster load times for the Ramdrive)

Gigabyte F2A55-M

Onboard graphics HD7660D with 1024 MB shared memory (works fine btw) (need to set that in BIOS options, if it only has 32MB to work with, than yes crappy)

2.7GB Free Memory under Windows XP-32(SP3) (3.2GB with new mainboard and seperate graphics card) (note also that if you set the shared memory to 512MB, most cases enough, on the old mainboard the available system ram goes up..) (personally, I'd rather stick to the 1GB shared memory with the 7660D, cuz better that than snailpace no ?)

12GB Ramdrive (Gavotte) with 2-4GB pagefile

If you want software to run really fast, install it on the Ramdrive, and before exiting, copy said intallment to the Harddrive..

I have a setup that does automatically loads two parts on Ramdrive.. Part one, being the low load parts, like browser (available roughly 2 seconds after Windows startup), and the bigger (game) loads secondly, which vary from a few seconds to almost 30 seconds..(with an indicator :D) If you wanna savescum through certain parts of a game and are tired of waiting -30 seconds or so per re-load, well, this setup does it in like 3 seconds..yes..that much faster..

Also, one more thing of note: Windows XP runs Windows XP software best.. So, despite VM-ing and all that jazz on newer systems, yes, still suing windows XP-32 in 2017 works best for my gaming, since I can also use windows 7 for newer stuff..

UGH.still more info though..if the BIOS or OS somehow makes part use of the >4GB ram, then Gavotte may fail, and will halt the system upon rebooting once the Ramdrive gets installed.. Make sure to try a (throwaway) PCI card when that happens, and switching the onboard GPU off and setting it's memory use to 0. i had that trouble on my newer mainboard, but after installing the add-on graphics card, it worke again..I still miss my old mainboard.. (got fried when power unit failed and sent 220 through it.. D: ) (yes using old power units may be hazardous to system) (but hey it was salvaged right ?..really bawd..)

Good luck and much fun gaming at extreme fast speeds and load times.. (for those saying Windows XP-Pro SP3, bawd, try getting nearly 800 games to work on newer OS'es..SOOOO not fun..) (also HD7660D internal graphics are SOOO not crappy, specially for older software, but make sure to install correct driver..) (in most cases 14.4 will work, but sometimes that driver is too new..) (yes, it's possible for that being too new, since some games check for stuff first, and then they can't recognize either the CPU or GPu or both..) (it needs to say the CPU specs under the My Computer (right click)..)