18

I wrote a .NET 4.5 application that buffers colour, infrared, and depth data from a Kinect v2, performs some processing on it, and then dumps it to disk, in uncompressed form; the .NET application also starts ffmpeg as a subprocess and pipes colour data to it to be encoded as H.264.

Because I'm not using an SSD, the video data arrives quicker than I can write to disk. But that's ok, it's acceptable for me to discard video frames when I'm low on RAM. My only requirement is that whatever I keep be mostly contiguous 8- to 10-second chunks of video. So I have added some logic in my .NET 4.5 application to start discarding video frames when I don't have enough RAM to buffer contiguous 8 to 10 seconds of video (roughly 1.5 to 2 GB).

And, to prevent page thrashing, I have completely disabled paging files. This leaves me with a total of 16 GB physical RAM.

My problem is that even with that mechanism in place, sometimes my .NET application or the ffmpeg subprocess still get killed when Windows 8.1 freaks out about low RAM, because obviously my application is using the most RAM when it has a huge backlog of video data to write to disk. Is there a way to tell Windows that my processes are more important than others so that Windows would start killing other less important processes first?

  • 10
    I didn't think windows killed processes, I thought that was a linux only feature. – Scott Chamberlain Dec 18 '14 at 3:58
  • 4
    @ScottChamberlain: That's because turning off the paging file on Windows is very rare. It gets you all kinds of unexpected and unusual behavior. The obvious answer here is "don't turn off the paging file; that forces Windows to keep unused data in RAM so your app can't use that RAM" – MSalters Dec 18 '14 at 13:44
  • 1
    If this was a StackOverflow question, I could point you to CreateMemoryResourceNotification which is a lot less hacky. – MSalters Dec 18 '14 at 13:49
  • 7
    @Kal: If disk access is a bottleneck, use a stronger compression, if CPU is a bottleneck, use a faster compression. If both are a bottleneck, rethink your entire design and start over, or get better hardware. – Mooing Duck Dec 18 '14 at 20:26
  • 1
    @FactorMystic OMG he did what? Disabling page file is going to reduce your usable RAM significantly. – Aron Dec 19 '14 at 3:12
45

Windows doesn't kill processes when all of the RAM is used. What actually happens is that processes fail to allocate memory and crash.

This is happening because all of your physical memory is in use and because the pagefile is disabled, the memory manager no longer has the ability to write pages that are not being used. This keeps your physical RAM full and when your process, or anything else running at the time, tries to allocate a page, it fails. Some applications crash.

This presentation from Technet explains: http://channel9.msdn.com/Events/TechEd/NorthAmerica/2011/WCL405

The pagefile is keeping applications from crashing when you utilize all of your memory by acting as a backstop for the over commitment.

Virtual memory is pretty much the foundation of how modern operating systems allocate resources, so it's all about having things in RAM that are in use, and moving stuff in and out from disk.

There are really only two answers:

  1. Re-enable the pagefile and increase the RAM on your computer to reduce disk thrashing.
  2. Reduce the memory requirements of your application.

The bottom line is that RAM is just another level of cache, and all of the stuff about virtual memory, pagefiles, memory mapped files, and all that basically comes down to this: if you're running out of memory, you need to add more.

  • 4
    Or use less.... – nhgrif Dec 18 '14 at 12:18
  • 1
    Please note that the backlog is building up because the data can't be written to disk fast enough. I don't think that enabling virtual memory on the very same disk can help there... – Alexander Dec 18 '14 at 12:41
  • 3
    In fact, the page file will be somewhere else on the disk. And since we know it's not an SSD, that means a physical seek which is the slowest disk operation. – MSalters Dec 18 '14 at 13:46
  • 9
    Sounds like you need explicit memory management in your application then... – Joe Dec 18 '14 at 15:16
  • 1
    @Joe exactly this. The garbage collector is going to make memory management a nightmare in these type of situations. This type of situation is trivial for me to deal with in C++ because I have fine-tuned control of all memory usage. Though there are design patterns that will work just fine for this case in C# as well, it's not as simple as what most people would try for. – Thebluefish Dec 18 '14 at 20:03
0

Go thru Windows Tool Panel & Advanced Settings & disable unneeded things, like window effects if you haven't already, and get Sysinternals Process Explorer &/or System Monitor to find & turn off anything extraneous that's wasting CPU or memory.

More importantly, use Process Explorer &/or System Monitor to watch as your program is executing and see exactly where & how it fails. Which thread runs short of memory and dies first - the main prgm or the ffmpeg part? Is there a specific dll or other shared resource that balloons unexpectedly in size? Or is the execution proceeding correctly, except biting off more than it can chew in data?

Figuring out more precisely the nature of your problem will likely point you in the direction of a solution. You could, for instance, implement your frame dropping policy more aggressively, while optimizing better for your 8-10 sec chunk criterion to achieve lower overall RAM overhead

Final suggestions: Maybe consider switching to Linux, and in the meantime, re-enable the paging file (linux calls it the swap space, which makes it sound more fun IMHO, like a swap-meet or something!) Good luck.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.