12

Does Linux have a mechanism to "scrub" memory? e.g. testing the memory and marking areas as dirty if they fail so that the system can continue to operate "safely" even with bad ram chips installed?!

4 Answers 4

3

The answer is yes, and it is done transparently (provided you have ECC memory to detect errors, and your kernel version is at least 2.6.30 to continue to operate safely).

Basically, your memory is checked at every read from the processor, and scrubbed periodically*, to check for consistency with the Error Correcting Codes (ECC). If an error happened, you get a Machine Check Exception, which is then logged and grabbed by mcelog (http://www.mcelog.org/).

If your error was correctable, it increments a "leaky bucket" counter, which causes a physical DIMM that fails too often to be transparently replaced by another one. Thus your memory page is copied to a new location, your virtual memory address is updated to point to the new page, and the old page is marked by the OS as not to be used anymore.

This is called "soft-offlining" on Linux (and memory page retirement on Solaris, I don't know about other OSs).

If your error was not correctable however, what is called "hard-offlining" happens, that is your memory page gets removed from the normal operating system memory management, and your application gets killed (NB : by some catchable SIGBUS signal that tells you where the error happened, but it's rare enough not to care and try to catch it). If your memory page is mapped from a file and clean, the OS can also reload it transparently at another physical location instead of killing the process.

You can read more on mcelog, there are plenty configuration options, you can get other behaviours to be triggered, options, and other leads on what to read and how to make sure mcelog is running on your system.


* Scrubbing, or "Patrol Scrubbing" consists in reading memory, checking it against ECC for errors, and overwriting with the corrected memory words when an error is discovered. The term patrol scrubbing is used by opposition to overwriting incorrect data on errors in memory reads, which is sometimes called "Demand Scrubbing". Scrubbing is a hardware procedure that can be enabled, usually through the BIOS.

7
  • 1
    This only applies if you have the more expensive ECC memory.
    – psusi
    Dec 25, 2014 at 15:44
  • 1
    This applies to all memories with ECC. Be it parity (but then you can't correct), SECDED, the more expensive Chipkill or any newer ones. DDR1 could already implement ECC, but it would all depend on which actual model you use. The "home" market has traditionally less need for resilience, but supercomputers have been equipped with them for over 20 years -- servers are in between.
    – Cimbali
    Dec 25, 2014 at 16:01
  • 1
    I meant ECC memory is more expensive ( than non ECC ) and so most people don't have it.
    – psusi
    Dec 25, 2014 at 16:16
  • 1
    Well "most people" is pretty vague. Whether it is common to pay the price in investment and power depends on the market, as I said. My average Dell laptop, that's 2 years old now, is equipped with it (standard, no special options asked). It's getting more and more common, because miniaturization of features makes DIMMs more sensible to various radiations.
    – Cimbali
    Dec 25, 2014 at 16:30
  • 1
    Cimbali, who do "Patrol Scrubbing" (on systems with ECC memory) - bios firmware (probably in smm mode, transparently for OS kernel) or linux kernel in some software mode (which module do the patrol scrubbing)? ECC memory did not check ecc sums; to check ecc, data must be read (and ecc scheme in memory controller will check sum). Some memory is read often (by normal programs on cpu), another may be not read for weeks. Patrol scrubbing will read all memory every day (intel) or every 1-48 hours to do ecc checking - electronics.stackexchange.com/q/73546#comment911379_73573
    – osgx
    May 25, 2018 at 3:01
7

This is actually a bad idea. Memory cannot be reliably tested in a quick sweep. This is why software like memtest86 uses multiple passes with different bit patters to test memory. Solution:

  1. Test memory with memtest86, preferably long test, leave it running overnight, it will take a long time.

  2. If bad memory is detected, use memmap kernel parameter to force kernel not to use that memory:

   memmap=nn[KMG]$ss[KMG]
            [KNL,ACPI] Mark specific memory as reserved.
            Region of memory to be used, from ss to ss+nn.
            Example: Exclude memory from 0x18690000-0x1869ffff
                     memmap=64K$0x18690000
                     or
                     memmap=0x10000$0x18690000

In addition, you can use ECC memory which will correct 1-bit errors and detect 2-bit errors in your memory automatically (and you'll get log messages from kernel about uncorrectable memory problems if they happen)

7
  • Thanks for the tip on those kernel parameters. Do you think you please can clarify why this is a such a bad idea and why you can't check a chunk of memory using the same methods as memtest86(+)? I am aware that more reliable testing requires more CPU time (and probably bigger chunks of ram in one go as well) but why would this have to be a show stopper? CPU time may not a problem if spread over a long enough period and besides multi cpu is becomming more and more mainstream.
    – Waxhead
    Dec 28, 2011 at 21:32
  • Well technically, if done over long enough period of time, this may be possible. But the bottleneck here is not CPU(s), but memory bus, and of course you "poison" your CPU's memory cache. I'd not aware of such kernel module, and the idea looks very fragile to me (orchestrating repeated pattern writing to an arbitrary region of memory on a live system, etc.)
    – haimg
    Dec 28, 2011 at 21:50
  • haimg : question : will the VFS manage paging for this reserved memory ? i think it can not as it won't be visible to it.
    – Jay D
    Sep 19, 2012 at 22:30
  • 1
    @Waxhead Memory scrubbing is usually done at the BIOS level using hardware. If enabled you should find options for patrol scrubbing and demand scrubbing. If memory integrity is important to you, which it surely is if you're using ECC memory, then the smallish performance hit gained by enabling these options is worthwhile.
    – Ian
    Sep 23, 2013 at 16:49
  • 1
    I think it could be interesting to include a reference to the badram kernel module here. It uses memtest86 as you propose, but instead of refraining the kernel from using bad memory, it allocates it to the kernel to not using, effectively guaranteeing that neither the kernel nor your applications run into that memory.
    – Cimbali
    Aug 9, 2015 at 11:06
2

The post and answer misunderstand the issue. Memory scrubbing is intended to keep correctible single bit errors from turning into uncorrectible double errors. The scrubber merely all physical memory (forcing cache misses to do so) occasionally. If there are any single bit errors, they will be corrected (and the correction must rewrite the correct value using a compare-and-swap), thus clearing the error.

Otherwise, if a second error occurs in a word which already has one error, the entire word will be uncorrectible and the OS will have to do something drastic.

Scrubbing is important because without it, memory which is read but not written (like code pages) may accumulate errors over time.

3
  • Why do you think the answer miss understood the issue when it has been marked as the answer?
    – Dave
    Jun 13, 2013 at 14:35
  • 1
    Notwithstanding Dave's reply, Larry is quite correct, the answer /does/ misunderstand the question. The question asks if linux can do a memory scrub, used, as Larry carefully explains, to prevent single bit errors detected and corrected by ECC h/w from turning into uncorrectable 2 bit errors. The answer talks about how to detect those errors in the first place using a software application.
    – Ian
    Sep 23, 2013 at 16:49
  • I think you misunderstand the purpose here. You are of course correct in your description about scrubbing however if you for example run a (non-critical) file server on non-ecc ram and have CPU cycles to spare it sounds like a good idea to sooner or later be able to detect corrupt memory and flag it as bad and know about it than to be unaware of a bad memory chip. Perhaps a better wording would be memory validation / verification. Not technically scrubbing perhaps, but still a viable way of reducing the damage done by potentially bad memory.
    – Waxhead
    Feb 19, 2014 at 22:31
1

If you have ECC memory you may want to have a closer look at https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-devices-edac. (I found "sdram_scrub_rate" especially interesting.)

(If this link breaks at some point (it really shouldn't) I'd suggest downloading the appropriate Linux documentation and search for "scrub".)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .