Setting /proc/sys/vm/overcommit_memory to 1 and then exhausting addresses

Question

As I understand it, setting /proc/sys/vm/overcommit_memory to 1 is supposed to make sure malloc always succeeds, and set the OOM killer loose if there's an actual memory problem.

I'm wondering what happens when you've malloc'd so much memory that you've exhausted the address space for your process? Does it return NULL despite the overcommit_memory setting, or does your process get a signal? Or something else entirely?

Answer 1

Malloc should return NULL and set errno if data segment runs out (even with overcommit enabled).

See manual for system call brk(2). Malloc calls brk to extend data segment. brk fails and sets errno to ENOMEM if extending the segment fails for any reason.