Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have the following problem: A program has a bug like the following

int main() {
  for(;;) {
    char *p = (char*)std::malloc(1024);
    std::memset(p, 1, 1024);

And it keeps allocating memory all the way, until my system starts swapping pages of other applications out in favor of that program and i can't do anything anymore on the box. I've hit this problem several times with different applications (today, it was with moonlight 2 beta in firefox). I though the problem is because the program causes other program's memory to be swapped out, and so it could use more physical memory.

Naturally i looked into ulimit, and found two settings

-m the maximum resident set size
-v the size of virtual memory

I read that the first denotes the total physical memory size the process can use at once. To me, it seems that this is more sensible than the total virtual memory size, because it may be shared, and it could be of no importance at all because it's swapped out anyway. So i added the following to my .bashrc after looking at usual resident set sizes with top, which range up to around 120MB for a usual firefox session, i found.

# limit usage to 256MB physical memory out of 1GB
ulimit -m 262144

But after running my above test snippet, it still brough my system down, and i had to wait around 5 minutes until the terminal recognized my ^C key presses. Usually, if i don't react within the first few seconds, in these situations i can only press the reset button, which i really don't like - so does anyone have a strategy for how to solve this? Why doesn't the physical limiting work? It seems to me that this way other applications should still have enough physical memory to react sensibly.

share|improve this question

Did you try with the -v flag?

The resident work set is defined by the maximum amount of memory kept as a working set in RAM before swapping. Thus it won't limit the total amount of memory inclusive swapped parts of the working set. The -v flag should do the job.

VIRTUAL MEMORY SIZE: the most useful of the memory-related limitations, because it includes all types of memory, including the stack, the heap, and memory-mapped files. Attempts to allocate memory in excess of this limit will fail with an out-of-memory error.

It is quite astonishing that the resources in the net describing this in depth are not really easy to find! I made the experiment on my linux box.

It stops with exception with

  ulimit -v 100000

and did not with the -m flag. The only thing that surprised me is the exception:

segmentation fault

I would have expected out of memory.

share|improve this answer
malloc() will not raise an exception, but return zero. Then the code will try to write there... so its segmentation fault. – liori Oct 3 '09 at 0:15
However i wonder why my system gets stuck even with -m. I thought that if the physical memory is limited for process (its working set), then other applications won't be swapped out and are still responsive. But the system locks down anyway. Why is this? If nothing else helps, i will use virtual memory though, even though that includes many things that don't participate in bringing the machine down (i.e memory mapped files etc..) – Johannes Schaub - litb Oct 3 '09 at 1:16
@liori, good point, dumb me, I think I'm doing too much C++... :-) – jdehaan Oct 3 '09 at 6:37
@litb I meant that with -m the amount of physical memory used is limited but not the virtual. So the process will swap. I read that not all flavors of Shell/Linux support -m. You could verify what is actually set with ulimit -a or with ulimit -m (without parameter) – jdehaan Oct 3 '09 at 6:52

I think ulimit implementation sucks randomly on almost any operating system. I've been using ulimit -S -v on Linux for six years. Six! And it seems that some kernels are not supporting it anymore. I got a Mac and OS X 10.6 is not supporting it. I've also tried with -m. In my old desktop with Ubuntu 9.04, ulimit -S -v works fine.

Indeed, I'm looking for a real solution to this. I just cannot believe people in Apple, for example, allow their program to go wild on memory.

share|improve this answer

Classic memory leak. I'm afraid, you need to bugfix the application in question, there's not much the operating system can do. All the operating system could do is restrict the memory for one application, but then the application in question would crash/segfault, and perhaps take the operating system or at least xorg with it.

You could ld_preload another version of malloc, and add a thread to free it after a specified amount of seconds. I doubt that it would work, though...

You would need this tut:

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .