4

Linux OS:es will typically allow overcommit of memory. For example a process can allocate 100GB memory, even though the machine only has 8GB physical memory and no swap.

As long as all the allocated memory isn't actually used, the process will work. If the program tries to use it all, the OOM-killer will kill processes to free up memory.

How does this work in Windows? Will Windows refuse to give processes virtual memory unless it can guarantee that this memory can be backed by actual memory (physical RAM or swap)?

5
  • 2
    Historically Windows has never allowed overcommit and as far as I know this has not changed in Windows 10. Overcommit has it's advantages but also some potentially serious problems. It can cause a process that has done nothing wrong and made only modest memory allocations to crash with no possibility of recovery. Windows designers took the more cautions decision of never allowing overcommit.
    – LMiller7
    Mar 31 '17 at 18:17
  • Thanks! Do you know if this is documented somewhere?
    – avl_sweden
    Apr 1 '17 at 22:36
  • 2
    The Microsoft publication "Windows Internals" 6th edition, covering up to Windows 7, describes memory management. Overcommit is not supported. The 7th edition, covering Windows 10, I believe is not yet released. But I am sure as big a change as overcommit would have been mentioned somewhere. These books can be tough reading if you do not have the background.
    – LMiller7
    Apr 1 '17 at 22:59
  • Thanks again! Maybe you could put this as an answer so I can accept it?
    – avl_sweden
    Apr 3 '17 at 7:53
  • @avl_sweden you can write an answer youself. This will at least allow closing duplicate questions. Jul 7 '17 at 18:14
6

Answering my own question, as no one else has.

It seems to be the case that Windows will NOT overcommit memory. This is actually a big difference compared to Linux.

Windows will allow a program to allocate more (virtual) memory than there is RAM on the machine, but ONLY if there is enough free disk space to be able to back the virtual memory requested by the program by disk if necessary.

4
  • 4
    Yes, exactly. The Windows "commit limit" is simply the RAM size + the current pagefile size. Windows won't let applications + the OS commit more than that. If pagefile expansion is enabled and a commit attempt would take you above the current limit (but less than would be allowed by the limits on pagefile expansion) the pagefile is automatically expanded to allow the request. Jul 21 '17 at 3:47
  • It still needs an OOM killer though, as mine crashed multiple times the past few weeks due to presumably dynamically assigning all my 10 GB free HDD space to Chrome. Feb 6 '18 at 15:52
  • Windows is overcommitting memory for stacks! Oct 1 '19 at 5:45
  • Oh, that's interesting! Do you have a link or other reference for this information?
    – avl_sweden
    Oct 2 '19 at 7:15
4

Here, the proof that Windows is overcommitting stacks.

Complile this with MSVC / 64 bit:

#include <Windows.h>
#include <iostream>
#include <cstdint>

using namespace std;

DWORD WINAPI stackThread( LPVOID lpvThreadParam );

#pragma warning(disable: 6387) // parameter ... could ne null

int main()
{
    HANDLE hThread = CreateThread( nullptr, 0, stackThread, nullptr, 0, nullptr );
    WaitForSingleObject( hThread, INFINITE );
    CloseHandle( hThread );
    return 0;
}

char *__fastcall getSp();
char *__fastcall getTlsStackLimit();

DWORD WINAPI stackThread( LPVOID lpvThreadParam )
{
    char          *stackTop,
                  *stackBottom;
    char          *curStack   = getSp();
    char          *tlsSp      = getTlsStackLimit(),
                  *efSp       = nullptr,
                  *ehSp       = nullptr,
                  *tlsSpAfter;
    SYSTEM_INFO    si;
    char volatile *p;
    GetCurrentThreadStackLimits( &(ULONG_PTR &)stackBottom, &(ULONG_PTR &)stackTop );
    GetSystemInfo( &si );
    for( p = stackBottom; ; )
        __try
        {
            *p;
            break;
        }
        __except( efSp = getSp(), EXCEPTION_EXECUTE_HANDLER )
        {
            p    += si.dwPageSize;
            ehSp  = getSp();
        }
    tlsSpAfter = getTlsStackLimit();
    (intptr_t &)p     &= -(intptr_t)si.dwPageSize;
    (intptr_t &)tlsSp &= -(intptr_t)si.dwPageSize;
    (intptr_t &)efSp  &= -(intptr_t)si.dwPageSize;
    (intptr_t &)ehSp  &= -(intptr_t)si.dwPageSize;
    cout << "(base for pages is one)" << endl;
    cout << "guard page hit page:    " << (stackTop - p)          / si.dwPageSize << endl;
    cout << "start of tls page:      " << (stackTop - tlsSp)      / si.dwPageSize << endl;
    cout << "exception-filter page:  " << (stackTop - efSp)       / si.dwPageSize << endl;
    cout << "exception-handler page: " << (stackTop - ehSp)       / si.dwPageSize << endl;
    cout << "tls page after gp hit:  " << (stackTop - tlsSpAfter) / si.dwPageSize << endl;
    return 123;
}

and this with MASM

PUBLIC ?getSp@@YAPEADXZ
PUBLIC ?getTlsStackLimit@@YAPEADXZ

_TEXT SEGMENT
?getSp@@YAPEADXZ PROC
    lea     rax, [rsp + 8]
    ret
?getSp@@YAPEADXZ ENDP
_TEXT ENDS

_TEXT   SEGMENT
?getTlsStackLimit@@YAPEADXZ PROC
    mov     rax, gs:[010h]
    ret 0
?getTlsStackLimit@@YAPEADXZ ENDP
_TEXT   ENDS

END

And link it together as a 64 bitt application. First, the curent stack-pointer is taken. Then, the current stack bottom is taken from the thread-environment-block (GS:[010h]). Then the lowest stack address and the address of the first byte after the stack is taken and the page-size is determined. Then the stack is touched from the bottom up to the guard-page. The guard-page will be hit very nearly to the actual stack-pointer and the stack-bottom which was determined from the thread-envinronment-block. After that is pointer is determined again to test if it has been updated by the kernel when the guard-page has been hit. Then the page-indices (1-based) of the guard-page, of the stack-bottom told by the thread-environment-block, of the SEH filter-function, the SEH-handler and the stack-bottom told by the thread-environment-block after the guard-page has hit are shown.

6
  • For readers who don't want to actually compile and run the program, can you describe what happens when it is run, and why this means windows is overcommiting memory for stacks?(The downvote was not from me)
    – avl_sweden
    Apr 29 '20 at 19:18
  • @avl_sweden ??? I described it under the sourcecode. May 1 '20 at 2:16
  • 1
    Ah, I read that as a description of what the program does, not what happens. What happens when windows cannot serve more memory to the program, after the alleged overcommit? Is the program terminated? What I want to know is the output from your program.
    – avl_sweden
    May 2 '20 at 8:44
  • 1
    @avl_sweden: You'll get a guard-page exception in the thread for which no more stack-space could be allocated. May 3 '20 at 9:04
  • Thank you, that was the information I was missing. So windows allows a program to set a stack limit which is larger than the ammount of virtual memory actually available? Interesting!
    – avl_sweden
    May 4 '20 at 10:32
0

Not sure if this is relevant, but certain applications will have their memory use limited, like Powershell, and throw these errors unless memory allocation settings are changed. Or you run commands through a separate application, like Spyder.

3
  • I wrote a test-program that did a lot of VirtualAlloc-allocations on one side - with these the memory-usage growed -, and created a lot of threads with 1mb default-reservation - and the System-commit grew only as large as the thread's initial stack-pages. Dec 7 '19 at 17:35
  • @BonitaMontero Yeah, but did you watch the size of the windows "swap file" on disk? Windows does actually reserve all that memory, even if it doesn't show up as committed, if I recall correctly.
    – avl_sweden
    May 2 '20 at 8:45
  • The system-commit didn't grow, and thereby the swapfile didn'g grow. May 17 '20 at 5:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.