In addition to the other answers, you can configure Linux to require backing for any allocated memory, even if programs don't use it.
Overcommitting memory and fearing the OOM killer are not necessary parts of the Linux experience, however. Simply setting the sysctl parameter vm/overcommit_memory to 2 turns off the overcommit behavior and keeps the OOM killer forever at bay. Most modern systems should have enough disk space to provide an ample swap file for most situations. Rather than trying to keep pet processes from being killed when overcommitted memory runs out, it might be easier just to avoid the situation altogether. [Respite from the OOM killer]
If a program allocates memory, the kernel can simply mark more pages of swap as committed. This indication is stored in the kernel's memory manager, the actual disk space isn't touched yet. Until that memory is used, nothing actually has to be swapped in and out. If they are never used, then swap usage will fluctuate without impacting performance.
Because processes are presented with their own address space or "view" (this is how swap works in the first place), the kernel has a lot of leeway in how it manages that. Using a fork example also from the article linked above, since it's much more likely to have shared memory pages than it is to freshly allocate a large amount of unused memory, memory can be allocated copy-on-write, increasing the swap use count. When it's actually written to (which might not happen), then that "committed swap" can be replaced with any unused RAM (then increasing RAM use and decreasing swap use). Imagine a process with 500MB allocated which forks on a machine with all or almost all RAM in use. If there is 500MB available in swap (and disk space is cheap, how big is 1% of today's TB drives? :P), no memory has to be copied (yet, and possibly never), but the kernel can guarantee those allocations are "successful" and continue to use the shared memory pages for as long as possible.
Thus the possibility of the OOM killer is avoided, and it's much simpler to design most software with the assumption that memory allocations (including "implicit" allocations through something like fork) either succeed or fail immediately, with the practical realization that if memory must be swapped then it might impact performance. That impact is almost always slight, but in the worst case leads to swap thrashing (still sometimes preferable to an outright kernel crash or OOM killer).
Though I don't know the exact details of how the Linux memory manager works, this answer is my own generalized understanding and what I remember reading over the years. I've tried to re-edit this answer so a minimal understanding of OS design is required (it's considerably complex and not something I'm terribly interested in myself), but it seems to ramble a bit; please let me know if you see how it could be improved. On the gripping hand, it might not be such an embarrassingly basic question.