60

A decade or two ago I could buy ECC (Error Correction Code) RAM for PCs I assembled. ECC RAM provided SEC-DED, I guess from bit flips caused by ionizing radiation (I don't know what else could cause transient bit errors to pop up in RAM or I/O buses).

I haven't seen ECC RAM offered for PCs for years now. Why is that?

If ECC memory was helpful twenty years ago presumably it would be more helpful now that PCs are running with 1-2 orders of magnitude more memory, at lower voltages and with smaller physical features that (presumably) are more susceptible to corruption from stray radiation. Are any of these assumptions incorrect?

I.e., if ECC RAM was considered a useful feature a decade ago, do the reasons it was useful no longer apply to current personal computers and servers? Or is the thinking now that ECC RAM was never actually useful?

5
  • Presumably the people who make money by building and selling hardware decided they could make more profit by eliminating the cost of supporting ECC on the motherboard... – Darwin von Corax Mar 22 at 3:22
  • 1
    Most server boards and Macs support ECC RAM (most routers also have ECC RAM) - it's usually only desktop and laptop motherboards that don't and simply because a CPU may support it does not mean the motherboard will. There is a case to be made that ECC RAM isn't vital to consumer systems (it would be nice to have, but it's not vital, and not having it is more of an inconvenience), whereas it is within server and router OSes where problems can compound exponentially from corruption – JW0914 Mar 22 at 13:27
  • 4
    I have first-hand experience of a single-bit undetected error in non-ECC RAM. See this question and the accepted answer. – Ex Umbris Mar 22 at 20:13
  • 14
    it is still standard on servers, and workstations. – Florian Heigl Mar 23 at 14:45
  • 9
    "current personal computers and servers" really? Every server I worked with (except for one which was just a renamed tower PC) had ECC memory. – TooTea Mar 23 at 14:53
37

A decade or two ago I could buy ECC (Error Correction Code) RAM for PCs I assembled. ECC RAM provided SEC-DED, I guess from bit flips caused by ionizing radiation (I don't know what else could cause transient bit errors to pop up in RAM or I/O buses).

There are 3 general causes of bit errors, the first two of which are single event upsets:

  1. Radiation (primarily free neutrons). This particular phenomenon is dependent on a number of things such as the neutron cross section of the particular device. It may seem counter-intuitive, but the newer much smaller geometries have a lower probability of an upset due to neutrons because they have been designed to be less susceptible. See the Xilinx link (from below).

  2. Lead, specifically Pb210 which is part of the Uranium decay chain and is found in older kit in the balls of BGA devices. Xilinx refers to errors from this as the alpha rate as they emit an alpha particle during decay. Clearly not an issue for a great deal of current equipment that is lead free (but still quite an issue in aerospace where tin lead processing is still common).

  3. General bit error rate issues. A memory interface is a communication channel, and all communications channels have an error rate. Admittedly, you may never see a single bit error in the life of a particular piece of equipment as this is a statistical quantity. Errors due to electrical noise and poor device decoupling also fall into this category.

i.e., if ECC RAM was considered a useful feature a decade ago, do the reasons it was useful no longer apply to current personal computers and servers? Or is the thinking now that ECC RAM was never actually useful?

It was useful, but of limited value, although many side channel attacks can be mitigated by its use.

The real reason you can't find it in commercially available boards is simply cost and those boards that do have it have a rather large premium, far higher than the delta cost of the silicon to handle it and the extra 8 data bits (for a 64 bit memory system). The cost-benefit analysis doesn't support its broad availability.

I do remember a research paper from Boeing that discussed soft errors in a Denver data centre. The amount of free neutrons is (up to a certain level) proportional to altitude. The higher you go, the more there are.

If ECC memory was helpful twenty years ago presumably it would be more helpful now that PCs are running with 1-2 orders of magnitude more memory, at lower voltages and with smaller physical features that (presumably) are more susceptible to corruption from stray radiation. Are any of these assumptions incorrect?

The memory interfaces we have today are far more robust than you might think; for DDRx, the data strobes are differential (so they reject common mode noise) and lower transition voltages are actually better for high speed interfaces, as we proved years ago with ECL.

In avionics, and in particular flight safety critical avionics such as flight control computers, the use of ECC for L2 and beyond is mandatory as is the use of parity for L1. That is one of the reasons those cards are not from Intel or AMD.

[Update]. The specifics of just how memory cells are laid out has a rather large effect on their susceptibility to SEUs; Xilinx has taken a particular approach that effectively stacks memory cells in such a way that the probability of a high energy neutron causing a bit flip is significantly reduced.

As I am not an IC designer that is all I can really say. There is a great deal more information at the Rosetta Project.

8
  • 3
    >In avionics, and in particular flight safety critical avionics such as flight control computers, the use of ECC for L2 and beyond is mandatory For the same reason you mentioned for Denver: planes operate higher and thus experience more neutrinos on average. – JS. Mar 22 at 19:11
  • 2
    "Clearly not an issue for a great deal of current equipment that is lead free" Lead solder is still commonly used for die to package interconnect (RoHS Exemption 15, see also 7a, 24 and 34), which obviously has a great impact. You can see this in China RoHS declarations where there will be an X under Pb. The original RoHS ban applied primarily to package-to-PCB solders. – user71659 Mar 22 at 19:33
  • 7
    It's not just avionics. The vast majority of servers runs on ECC and so do workstations. It often makes little sense to save a couple hundred bucks on hardware (just a small fraction of the total cost) when it means risking losing thousands in lost productivity. – TooTea Mar 23 at 14:57
  • 5
    Modern Intel and AMD CPUs use ECC for their internal caches. (Possibly only parity for Intel's L1d, if Paul Clayton is right about that, because it has to support writing any individual byte without an RMW of the surrounding word to update a larger ECC granule. And ECC overhead is high for SEC-DED in 8-bit granules.) Internal cache SRAMs run at such low voltages that bit errors are more likely- they're more easily disturbed by noise. (realworldtech.com/near-threshold-voltage/2) – Peter Cordes Mar 23 at 17:02
  • 4
    You did not mention Row Hammer — accesses can disturb physically proximate memory cells (the capacitors are not perfectly isolated). Having the refresh rate too low for environmental conditions such as temperature can also introduce errors (and theoretically one could use ECC to save energy by reducing the refresh rate). – Paul A. Clayton Mar 23 at 17:08
62

15+ years ago Intel decided ECC RAM support was not of value in consumer machines.

In other words, Intel decided 15+ years ago that consumer machines don’t need it. Thus the market doesn’t support it outside of server hardware. Thus end consumers are paying the price.

This January 2021 article in ExtremeTech provides a fairly solid summary of what happened: “Linus Tovalds Blames Intel for Killing ECC RAM in Consumer Systems”:

“There was a time when you could buy ECC support on mainstream chipsets, but Intel phased out that capability on non-Xeon platforms a number of years ago. The 975X may have been the last consumer Intel platform to support it, and that family launched 15 years ago. The Xeon 3450 chipset was cross-compatible with certain high-end CPUs in the Nehalem family, but that’s still a Xeon chipset — not a mainstream part.”

“As a result, support for ECC in consumer products — and the availability of ECC RAM for consumer products — both fell off a cliff.”

Since the article quotes Linus Torvalds, here is his specific complaint:

“The memory manufacturers claim it’s because of economics and lower power. And they are lying bastards – let me once again point to row-hammer about how those problems have existed for several generations already, but these f*ckers happily sold broken hardware to consumers and claimed it was an ‘attack’, when it always was ‘we’re cutting corners.’”

The issue here is Linux is getting blamed for kernel errors, but Linus Torvalds believes the root cause are hardware issues that can be traced to the prevalence of non-ECC RAM in machines nowadays.

But that is a tangent… What it comes down to is PC manufacturers cutting corners. Classic manufacturing issue.

And nowadays where PC hardware is considered pretty disposable, there might be some rationale here: RAM starts to get flaky, just toss the machine and buy a new one. The truth is the market is filled with non-techs and non-PC builders so hey… It stinks but it is what it is.

6
  • 7
    There is a case to be made that ECC RAM isn't vital to consumer systems (it would be nice to have, but it's not vital, and not having it is more of an inconvenience), whereas it is within server and router OSes where problems can compound exponentially from corruption – JW0914 Mar 22 at 13:26
  • 4
    @JW0914 True, but look at the consumer market nowadays: It’s all disposable and not everyone has to be a “geek” to own a PC. 15 years ago was the cusp of consumer PC market really blowing up. So if someone — let’s say — is $150 to $300 for a PC, will they actually spend more $$$ for ECC RAM on a machine like that? Or will they just toss it when it starts to act “weird.” Also, ECC cost to the manufacturer might not ever reflect on consumer prices. Meaning, manufacturers could just be pocketing the $$$ saved from not having ECC functionality in RAM. It’s all crap for consumers. – Giacomo1968 Mar 22 at 15:07
  • 3
    I get the point you're making and one factor in why ECC is more expensive is due to lower demand (same reason DDR2 costs more than DDR4). Server & router OSes are coded to make use of ECC RAM, but are Windows 10 or popular Linux distros? It ultimately comes down to whether it's vital or not from a business-consumer standpoint (e.g. would consumers see a noticeable difference), as it's motherboard OEMs that control if a system can use ECC RAM (esp. when they're now soldering CPUs & RAM to the PCB), and it simply isn't vital to consumer systems in the same way it is to server & router OSes – JW0914 Mar 22 at 17:18
  • 9
    @JW0914 For Linux, the ‘distro’ generally has no involvement in ECC support other than enabling the kernel functionality at build time, and all sane distros do build their kernels with that functionality enabled because it has near zero cost in terms of runtime memory usage on systems where it’s not used, but is mission critical for most users who want it. – Austin Hemmelgarn Mar 22 at 17:47
  • 9
    @JW0914, ...Linux is one of the most common "server & router OSes". Anyhow -- if you've ever seen a system throw a machine check exception, ECC detecting (or correcting) an error is one of the most common triggers. – Charles Duffy Mar 23 at 15:32
38

I agree with the answer provided by @Giacomo1968 as far as history goes. The current state however is changing. AMD has recently started to support ECC memory in their current desktop CPU line for the AM4 socket: "ECC is not disabled. It works, but not validated for our consumer client platform." (Source: Reddit)

That said, the motherboard also needs to support this. Some consumer boards do, some don't.

3
  • 9
    Fascinating. Seems like AMD is trying to cater to the power-user/tweaker market. A market that is really getting smaller all the time, but one that is very loyal. – Giacomo1968 Mar 22 at 15:15
  • 3
    It's not something I've been following closely because I'm not planning to build a new system anytime soon, but from what I've seen in the Anandtech twitter feed per board support has been something of a mess with some boards cycling from actually using it to appearing to but not actually doing ECC seemingly at random between BIOS revisions. – Dan Is Fiddling By Firelight Mar 22 at 15:26
  • 7
    AMD has been supporting ECC on consumer desktop CPUs for years. The quote is an application of existing policy to a new socket, not a new policy. – Ben Voigt Mar 23 at 17:32
37

A bit more that addresses the question:

Intel unilaterally decided that consumers did not need ECC and decided to only provide it to server and workstation customers where Intel could charge a premium.

Microsoft tried to make ECC a required feature for Vista certification but Intel refused to do it. Before the Core i7 series the memory controller was part of the motherboard and ECC support was a motherboard chipset feature.

You can get laptops with ECC. For example, there's the Dell Precision Workstation line which you can get with a Xeon-W CPU and ECC RAM.

You can buy any Ryzen CPU. Well, any Ryzen without integrated graphics. For the integrated graphics to work with ECC you need a Pro version which is hard to find unless you buy it in a prebuilt system.

With a Ryzen and a motherboard like the ASUS PRO line the unbuffered ECC will work great.

For the registered, buffered ECC modules you need a real Xeon or EPYC CPU because those RAM types are controlled differently.

In the near future DDR5 RAM has the option to use ECC internally, without any notification or control from the CPU. It also has the option to provide signalling and control to CPUs that support it.

Here is an example of an ECC module you can buy today. I bought four of them for a Ryzen build:

"Crucial Server Memory 16GB DDR4 DIMM 288-pin - 2666 MHz / PC4-21300 - CL19-1.2 V - unbuffered - ECC CT16G4WFD8266"

I would provide an Amazon link but that may be considered spam. Also note that you can get ECC modules at 3,200 MHz speed today.

8
  • 1
    if buying/making a Ryzen system for ECC, be careful to research the parts first. My understanding is that even though some combinations of hardware "support" ECC RAM, that doesn't necessarily mean the ECC part can be enabled – anjama Mar 22 at 17:08
  • 5
    @anjama That is true. It depends on the motherboard and BIOS. Many ASUS boards support it and many ASRock boards. Other than those I haven't kept track. – Zan Lynx Mar 22 at 19:33
  • 1
    It is worth noting that Intel actually has the same silicon for different market segments, the segmentation is purely artificial. One and the same chip can become the desktop "Core-series" or a "Low Core Count" Xeon targetted at workstations/NAS, with features such as ECC enabled. Look out for Xeons with iGPUs. – BotOfWar Mar 23 at 18:54
  • 3
    I think the StackExchange policy is that you can link commercial products when appropriate, though there's an expectation to disclose affiliation. I've included Amazon.com links in a lot of my posts, e.g. this one. – Nat Mar 23 at 19:46
  • 1
    I think Amazon links are fine. But the issue with links to any catalog site is the chances of those links dying in the future are quite high. So the goal is to avoid having dead links but to provide as accurate information in the post as possible. And in the case of a product, try to find as solid a link as one can; such as linking to an official manufacturers website and such. – Giacomo1968 Mar 23 at 21:45
4

This is a reflection of consumer software quality, among other factors.

ECC RAM only really helps if the error rate from sporadic bit flips is the dominant cause of failures. If you run high reliability software which never crashes on its own, eliminating the few remaining sources of errors actually improves the MTBF of the system.

If you run software which is produced in a rush because of the "winner takes it all" economics, it will have plenty of failure sources besides RAM errors. Paying the premium to reduce your error rate by a few percent just doesn't make sense in this case.

And then comes the positive feedback typical for goods with high fixed costs: higher prices means less demand which in turn means even higher prices. I don't think Intel is to blame here: they didn't stop supporting ECC in consumer chips because they hate technology, they did so because they earn more money selling cheaper non-ECC chips.

Notably, in the world of industrial microcontrollers controllers which run software designed according to functional safety standards, ECC RAM is widely used today.

2

When talking about ECC or not ECC, discussions focus mainly on the bit flip problem. But in my 15 years of experience with more than 40 datacenter servers and/or home NASes, bit flip was not the biggest problem that ECC would have solved.

I had two big problems of possible or real data corruption, both were a totally fault RAM SIM module.

In the first case, having an ECC RAM module, the HP Proliant server just halted itself saying something like "ECC error, please replace the faulty module". No data was corrupted, despite the unexpected system shutdown of 8 virtual machines running on it.

In the second case, after the RAM module fault on non-ECC NAS, the NAS continued to write garbage data to the disks for some hours before crashing the OS. No data was recoverable for the last 24 hours of work. An entire virtual machine running inside that NAS was corrupted, and some other random files too. Everything has to be restored from the previous backup.

ECC is important. It not only saves you from a minor bit flip problem, but it also saves you from an entire RAM module fault which can destroy most of your data. I want ECC where I store data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.