27

Perhaps I may be overlooking some aspect that is an important cause as to why these don't exist, but I feel that having RAM expansions through PCIe would be perfectly feasible. I know that a lot of operating systems use virtual memory and store some lower priority items on hard disks, but considering the lower speeds I feel that we could use some bonus not-quite-as-fast memory.

6Gb/s SATA ~= 800MB/s
PCIe 2.0 = 500MB/s per lane. 16 Lane ~= 8GB/s

Sure, 8GB/s isn't as fast as actual RAM, but it's 10x the speed of SATA. Why not have a PCIe board with a couple of RAM slots for use with the old RAM that you just replaced with that recent upgrade? PCIe has the benefit of being on almost every motherboard out there. One 'adapter' PCIe RAM Expansion Board would be (supposedly) compatible most PCs.

What am I missing since this hasn't been done yet?

7
  • @rawbrawb, I think that is a SSD, not RAM. Feb 1, 2013 at 17:52
  • 4
    Note: Almost no PC have PCI-X. It was only found in server market and tops out at around 1 GB/s. Furthermore it's obsolete. It was development based on classical PCI and is not related to PCI Express (often abbreviated PCIe).
    – AndrejaKo
    Feb 1, 2013 at 20:31
  • 1
    Thank you for pointing that out AndrejaKo, I just assumed eXpress since I had seen PCIx before.
    – agweber
    Feb 4, 2013 at 16:09
  • 1
    Would be great to have like a "L2 RAM" -- something the OS could page stuff into if it needed -- i.e. for your DDR4 system, you could have a PCIe card that took your old DDR3 DIMMs and exposed them as an L2 RAM to the system... Sep 14, 2015 at 15:43

12 Answers 12

18

This is a complex issue, that is highly dependent on exactly what you want to do with that RAM.

In most cases, it is cheaper and better to simply replace the motherboard with a new motherboard that supports the amount of RAM that you require. I have a motherboard here in front of me that can take 16 memory modules. The largest module available is 32-Gig. That's a total of 512 Gigabytes in a single machine. (Never mind that 16 modules of that size would cost about US$14,000, or that the MoBo also has dual 8-core CPU's on it.)

Having the RAM on the MoBo means that it is the highest speed possible. You can use it for both a RAM-Disk as well as normal program and data storage. The best of both worlds.

But in your question you keep comparing it to SATA storage, so I am thinking that you'd want to use this extra RAM as a RAM-disk and not for general CPU RAM. This is a valid use, and years ago people did have PCI cards with lots of RAM on it specifically for this purpose. Those cards looked like another disk drive, and not just more CPU RAM. Often these cards had an external power connector on them so you could give them some sort of backup power in case the main power failed.

These types of cards have largely gone away. They were obsoleted mainly by three things: 1. Motherboards now can have much more RAM on them than in the past. 2. There are more modern solid-state drives using Flash memory and PCIe (some with large RAM caches) that work better. and 3. They were just too expensive for what limited advantages it gave.

There are other reasons why you might want to have a PCIe card with lots of RAM, but all of them are applications where the card is doing something other than just storing data. Like Video cards, or data acquisition cards. These things do not apply here.

7
  • 4
    Interesting. I've had a similar idea floating in the back of my head. The concept was a PCIe or perhaps SATA3 based "drive" that uses inexpensive sticks of last-generation ram, for volatile-only use. Common cases would be /tmp, swap, TempDB, and similar. It should be possible to obtain SSD-like read speeds, with much faster write speeds, and by using last-gen sticks it would cost substantially less than adding more general RAM. I suppose people find SSDs good-enough in most cases. Feb 1, 2013 at 20:14
  • 3
    Sure if putting that ram on the MOBO is viable. In a non-server environment, ram slots are often very scarce, and motherboard replacements are frequently non-viable. In a server environment things are rather different. Such a device would definitely have less impact there. I'll also admit that this would work better if prices for old generation RAM dropped faster than they actually do. Feb 1, 2013 at 22:10
  • 1
    @David - I'm sure that motherboard was very expensive though -- not to mention buying all that current gen RAM -- for the enthusiast consumer who always has last gen RAM laying around, having an "L2 RAM" card that the OS could use as swap (so it's slower, and maybe even only accessible serially, seriously not asking for much here -- but being able to reuse our old lastgen chips to speed our systems up would be great...) Sep 14, 2015 at 15:48
  • 1
    Heck -- even if it was RAM the OS couldn't directly address, it'd still be worth it (think of products like PrimoCache and Primo RamDisk that can use RAM the OS can't address, for example). -- You buy one card, and suddenly a bunch of extra RAM that you already have is available for third party programs to use to speed up your system... Sep 14, 2015 at 15:50
  • 1
    @KevinCathcart I agree -- one feature that would be killer is if had some flash rom on it, and you could program into it the "bad" areas, so you could use failed RAM for example -- sort of like what scandisk used to do for bad clusters (like, I have a couple of 8 GB sticks that have just 1 MB of failed area each -- add them all up, oh look, 31.9 GB...) :-) -- People always say, this would be too expensive of a card to build because it would need 2 layers per slot -- ridiculous -- if you expose it as a disk and access it serially, there's no need for that... (just use chip select). Sep 14, 2015 at 15:55
14

This has been done; many years ago you could buy ISA cards (pre-PCI) with RAM on, which presented to your PC as either "extended" or "expanded" memory. This was a way to get past the 1MB limit of the original PC.

Modern PCs have a section of extra RAM attached to the video card, separate from main memory.

The reason why you don't get RAM expansion cards nowadays is that latency is a serious problem. There isn't really any provision in the OS for preferred versus non-preferred RAM, so you'd have to use it as a swap disk / pagefile.

5
  • 7
    you could use RAM as a (non-permanent) disk too, as well as for swap. Feb 1, 2013 at 17:56
  • I remember having a 64k add-in card for my Apple IIe that doubled the available RAM to 128k. There was no separate video memory, so the add-on also enabled 80 column text and double-resolution 'graphics'.
    – HikeOnPast
    Feb 2, 2013 at 0:13
  • Yep: it's not the throughput that the problem. It's the latency. Feb 4, 2013 at 16:25
  • Yeah, those ISA cards sat right on the Processor's IO bus though -- no Northbridge / Southbridge -- it was basically as directly connected to the CPU as the onboard RAM was (at least, it was in my 286). -- I'm thinking, in a modern computer you could use PCI-e RAM as some kind of L2 RAM -- like you mention for swapping, etc. -- the OS doesn't even have to be aware of it (though, rightfully, it should). Sep 14, 2015 at 15:59
  • I'd just like to point out, Linux in fact uses their NUMA (Non Uniform Memory Access) support for this scenario. Typical NUMA has each CPU having it's pool of faster local memory. For what they call "tiered memory", where you have normal RAM plus "persistent RAM" (like Intel Optane where it's essentially extremely high write limit SSD), it adds some "NUMA nodes" with RAM that's marked slower and 0 local CPUs on the node; this tells Linux to use that memory for inactive stuff and put actively used stuff in the faster RAM.
    – hwertz
    Jul 4, 2023 at 6:39
6
+100

Update for 2021

This technology is arriving with the evolution of PCIe and DRAM, although we need to wait for the coming generations of both.

The AnandTech article Using a PCIe Slot to Install DRAM: New Samsung CXL.mem Expansion Module describes just this kind of technology being developed by Samsung:

Samsung’s unveiling today is of a CXL-attached module packed to the max with DDR5. It uses a full PCIe 5.0 x16 link, allowing for a theoretical bidirectional 32 GT/s, but with multiple TB of memory behind a buffer controller. In much the same way that companies like Samsung pack NAND into a U.2-sized form factor, with sufficient cooling, Samsung does the same here but with DRAM.

I'm a bit skeptical about the technology labels used here, especially PCIe 5.0 which might actually mean PCIe 4.0, as it sounds identical, and I don't think that Samsung will announce today hardware that will only be usable several years from now and for a standard that doesn't yet exist.

The article notes that a DRAM slot is usually rated maximum for ~18W. This means that a 2 TB add-in CXL module like this might need around 70-80 W, which would likely require active cooling and a big heatsink.

This kind of hardware is evidently destined for special-purpose computing, not for ordinary desktop usage.

enter image description here

1
  • woah this is neat!
    – tuskiomi
    Jun 21, 2021 at 5:06
2

Modern server systems achieve up to 75GB/sec between CPU and main memory and even mid-grade systems can support up to 768GB total DRAM capacity. Any requirement to scale beyond that with faster-than-SATA speeds is covered by FLASH PCIe solutions that boast x8 PCIe speeds and many TB of capacity without the data volatility issues associated with DRAM.

5
  • 3
    Nitpick: PCIe, not PCIx. They're two different standards. Feb 1, 2013 at 23:06
  • Valid. Corrected.
    – HikeOnPast
    Feb 2, 2013 at 0:08
  • 1
    Your "midgrade system" is a rack mounted server. Come on guys, we're talking enthusiast class. -- Current gen RAM is always super expensive, and we always have a bunch of last gen RAM sitting around. -- If someone built this product, they would make a damn killing... Sep 14, 2015 at 16:01
  • Your flash link is broken.
    – user10489
    Jun 19, 2021 at 6:16
  • 1
    Bill Gates was wrong when he said 640K was enough. Wasn't true then, and isn't true now for even larger numbers. The 4T some current server motherboards can take isn't enough, let alone 768G. There's never enough!!
    – user10489
    Jun 19, 2021 at 6:26
2

Some minimal support for that already existed in PCI: there is a class code/subcode 0x500, which reads "Memory Controller", and it is actually used for some internal devices for (slow) inter-device communication.

AFAIK the problem here is that cache coherency is not supported for PCI/PCIe devices in CPUs (and not required by PCIe standard). Without working cache for the memory range corresponding to PCI device you can say good bye to high speed -- instead of going to cache CPU would need to always go into PCIe device for any access to any given address. This problem was solved for CXL in CXL-supporting CPUs and devices.

1

Adding to pjc50's reasons, it wasn't successful with PCI cards either, see this discussion for more details.

With main memory being larger, there is also less of a need.

1

I've always wanted to do something about turning piles of obsolete left over ram into a ramdisk type of device. Unfortunately, with the way technology moves so fast these days, obsolete ram is typically at least an order of magnitude lower density and higher power consumption than system ram, so it doesn't make much sense even if it would be amusing.

The modern current equivalent of the ram card is Optane memory, which is cheaper and higher density (but slower) than the equivalent amount of regular ram. This is available both in PCI form factor and in standard DIMM form factor. Some versions of this can be dynamically repartitioned between acting like slow ram and acting like an extremely high speed disk. A few high end motherboards are designed to take Optane memory in regular ram slots. I'm not familiar with what kind of OS support is required to handle this, but the pci/sata/nvme versions just look like a regular disk.

1

There are PCIe ram expansions now, using CXL protocol. So it is CPU->PCIe slot->CXL memory controller + DDR4/5 memory. However such devices are still in development, no available product on market, only some companies have prototype parts. In 2024 there will be such products on the market, meanwhile some giant companies will start using such products in their servers.

1

It seems to me many people here are kinda missing the point of the question. Everyone seems to be thinking in terms of expanding RAM so they keep pointing out how slow it would be if used in an add-in card.

Yes, older RAM is slower and has more latency, yes SSD stuff is faster and non volatile, yes older RAM uses more power.

But RAM, in any of it's iterations is WAY faster then a hard disk and using it for the paging file will make a tremendous performance improvement of the system overall.

Also, paging files get used a LOT. Putting it on an SSD with limited write cycles is kinda wrong.

It's nice to think in terms of new mother boards with lots of fast RAM, but there are millions of people using older hardware that just can't afford to drop $1000 on a new computer. I know I can't. I have a lot of old RAM sticks just sitting around, it'd be nice to use them and you can buy that old stuff on eBay for pennies.

1
  • I think I covered this in my answer...I basically agree with you except you left out density. I looked into building a ram card for old ram, but the size of the old chips is just so small that it would be cheaper to buy more new ram than build a card of old ram that could support the 100 dimms needed to make a bank larger than main system ram. I suppose if you are using current chips, it might be more interesting, but I don't have those just laying around.
    – user10489
    Sep 27, 2023 at 22:00
0

i don't see how this is not possible to get close to ram speeds from nvme's i test triple channel ddr3 in a benchmark to 22GB/s

32x pci-e bandwidth isnt far from that. but

the ram has NANO seconds of latency while the ssd would have Millaseconds but thats also not a limitation of pci-e, thats purely current affordable retail storage. there is specialty non-volatile storage that could achieve this but the with the 50k price tag, i don't think retail would see this for atleast 10-15 years.

although, if you literally used ram modules on a custom board and the motherboard firmware would allow it to detect it as memory. technically, it would work to extend memory. the underlying tech is there for it to work. "<3 pci-e"

1
  • the problem with NVMe though is NAND (specifically) isn't built for mass writes... MRAM would be a solution, but sadly that's not made obtainable to consumers. but yeah NVMe == PCIe x4, so a custom solution could easily be made, though I'd more prefer direct PCIe x16 for more bandwidth (ZFS cache would greatly benefit from this)
    – Tcll
    Apr 7, 2020 at 0:32
0

The most recent incarnation of PCI RAM cards was produced by Gigabyte in 2005 and 2006 under the names i-RAM and GC-RAMDISK, respectively. They supported 4 slots totaling 4 GiB of DDR-400. The transfer rate was that of SATA, 150 Mb/s, and the latency of 0.1 ms.

3
  • Third name is ACARD ANS-9010 ddr2, near 2008. Jun 26, 2020 at 16:12
  • DDR3-1333 PC3-10600 (10666.67 MB/s max) + SATA III (600 MB/s max) + ZRAM CACHE (Linux) = 600 MB/s max ramddrive for swap+tmp+.cache, which is faster, then regular 80-160 BM/s hdd. Nobody even try make such solution public to regular end-users, because they don't want to pay for stateless ramdrive. Jun 26, 2020 at 16:29
  • DDR3-1333 PC3-10600 (10666.67 MB/s max, 2013) + SAS 4 (2812.5 MB/s max, 2017) = 2812.5 MB/s max, which is faster then DDR-333 PC-2700 (less 2700 MB/s max), near DDR2-400 PC2-3200 (near 3200 MB/s max, 2003), which is faster then regular SATA III SSD (550 MB/s max), so ssd is not fast for enterprise. P.S.: Calculations are approximate. Jun 26, 2020 at 16:51
0

I believe it is already stated that Gigabyte tried this and there were even attempts back in the 286 days. I've been around long enough to have bought and used both (I think I have the most recent version still on the shelf in my lab). The problem is every time it's been thought up designed, manufactured, and sold it, sales flopped. Last gen RAM is only guaranteed cheap if you already have it. After it becomes lest-gen there are unstable pricing curves that even go up instead of down. (That's totally aside from the bandwidth and latency issues that make it sub-par vs. actual RAM).

In the case of the Gigabyte iRam card, it happened to come out near the 32 to 64 bit erra, so one of it's drivers to market was offering some extra speed since the OS couldn't even address that storage without something qwerky like PAE (which really needed new programming methodology to be useful). With the current ceiling for addressing RAM up in the TBs, this leaves them with the option of creating cheaper motherboard upgrades and/or individual RAM stick price per GB improvements to shoot for.

The reality is that people seeking to eek out some extra performance can typically get that extra performance through some combination of CPU and/or motherboard (And thus RAM) expansion. If you can't buy the performance now, then waiting a little while often see what you want materialize.

I like how the OP thinks though, it's how I think, ways to skirt limitations quickly/easily, but sadly it's just not feasible. Until they find a way to present add on cards that are nearer and dearer to CPU performance, I don't think it's an option (and even there, manufactures have no way to make individual electrons achieve the latency of on-die and short range RAM traces, without drastically shrinking the way PCs are designed).

Cheers.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .