up vote 1 down vote favorite
share [g+] share [fb]

I have a program that requires a lot of memory. So much that the malloc fails as the system goes out of memory.

I don't care if the overhead for swapping pages in and out of the disk is too high - I just want the program to run. So, is it possible that by sufficiently increasing the swap partition size the out-of-memory error can be prevented by effectively making more memory is available?

link|improve this question
92% accept rate
Will someone please tell me how to strike out the word physical from my title. I don't want to remove it. – AnkurVj Sep 1 '11 at 6:44
3  
tried <s> and <del> - neither works. I think strikeouts in titles are disabled. – Journeyman Geek Sep 1 '11 at 6:52
feedback

3 Answers

up vote 3 down vote accepted

The malloc() function usually allocates virtual memory from the memory heap. Even though you have not specified an operating system nor programming language nor version of runtime library, I'm going to assume that your version of malloc() does not allocate physical memory. Therefore adding more physical memory or increasing swap space may not eliminate the out-of-memory error return you see from malloc().

For a solution, you need to determine if your program is exceeding the amount of (virtual) memory that is allocated, or if the system is actually running out of memory pages. "When Linux Runs Out of Memory" explains how malloc() allocates (virtual) memory, the two ways you might get an out-of-memory condition and how to determine which. Note that availability of memory pages is a complex combination of physical memory size, swap space size, memory usage and process load.

link|improve this answer
Okay, I thought that the amount of virtual memory that can be allocated by a program is limited by the total available memory (Physical memory + swap space). i.e. even if the amount of memory allocated to the memory heap maybe larger, malloc may still fail because there is no real memory left to allocate ! – AnkurVj Sep 1 '11 at 8:22
On linux, you could use the getrlimit(RLIMIT_AS, p) function to find out the maximum size of a process' total available (virtual) memory. If this limit is exceeded, the malloc() and mmap() functions shall fail with errno set to [ENOMEM]. You might want to also look at the setrlimit() man page. – sawdust Sep 1 '11 at 8:56
But even if the limit is not exceeded, can malloc fail because the system has no memory to allocate ? – AnkurVj Sep 1 '11 at 9:08
1  
Here's something to read about a process running out of memory versus Linux running out of swap space: linuxdevcenter.com/pub/a/linux/2006/11/30/… "The conclusion is that OOM happens for two technical reasons: 1. No more pages are available in the VM. 2. No more user address space is available. 3. Both #1 and #2." – sawdust Sep 1 '11 at 9:31
The link is indeed superb .. I'd recommend adding it to the answer – AnkurVj Sep 1 '11 at 9:53
feedback
up vote 1 down vote

Well, no. It does not increase your memory - it merely means that your OS is able to move things that aren't of immediate use out to free up memory. This will have no difference in terms of performance, but if the software in question isn't actively using memory, your system may be able to swap out RAM to your swap to use for other things.

You may want to try creating and using a swap file, as opposed to a swap partition to see if adding swap works.

You can also try to limit (though it may have consequences) the memory the software uses with ulimit.

Loads of swap is not a replacement for physical memory - you're either going to have to add more memory, or work out why your program needs that much memory, and fix it.

link|improve this answer
What I mean is that by sufficiently increasing swap space, can the out-of-memory error be prevented ? I don't care if the overhead for swapping pages in and out of the disk is too high. I just want the program to run. – AnkurVj Sep 1 '11 at 6:41
It should - assuming it dosen't slow down the system to a crawl from all the paging. My recommendation is a VERY large swap file to test, something like 5-8 times the size of your physical memory. – Journeyman Geek Sep 1 '11 at 6:48
feedback
up vote 0 down vote

Physical memory is, and only is, the memory chips inside your computer case. The swap file is used when you lack enough physical memory. Increasing the swap file size will help prevent out of memory errors, but I don't recommend it. Having too much swap file available will give your computer an unreasonable sense of ability and probably cause it to slow to a crawl when using most of the page file.

The best solution is to buy more memory chips. It'll make everything run that little bit better.

If you don't mind me asking, what is your application doing that consumes so much memory?

link|improve this answer
I am allocating space for a 60K X 60K matrix – AnkurVj Sep 1 '11 at 6:41
So, assuming 32-bit integers, you need 3.4Gb of memory? Ow! I'd suggest making your own swap file system, loading and saving chunks of the matrix as you need them. You can customise it to work in a way that serves you the fastest, unlike the OS's swap file which does what ever it does. – Hand-E-Food Sep 1 '11 at 23:16
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.