Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

The program is a memory exerciser that request multiple chunks of 512MB to malloc. For some reason the node 0 is been memory exhausted while the others are not been called. MemExerciser cpuset=/ mems_allowed=0-3 shows that it can access all 4 nodes of memory. There is no cpu or node affinity active at that time. OS is SLES11-SP1

[ 7332.854685] MemExerciser invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
[ 7332.854693] MemExerciser cpuset=/ mems_allowed=0-3
[ 7332.854697] Pid: 19168, comm: MemExerciser Tainted: G          NX 2.6.32.36-0.5-default #1
[ 7332.854699] Call Trace:
[ 7332.854720]  [<ffffffff810060ac>] dump_trace+0x6c/0x2d0
[ 7332.854731]  [<ffffffff813aab68>] dump_stack+0x69/0x71
[ 7332.854741]  [<ffffffff810b593d>] oom_kill_process+0xcd/0x2f0
[ 7332.854747]  [<ffffffff810b5fb0>] __out_of_memory+0x50/0xa0
[ 7332.854752]  [<ffffffff810b6198>] out_of_memory+0x198/0x210
[ 7332.854758]  [<ffffffff810b966b>] __alloc_pages_nodemask+0x64b/0x660
[ 7332.854767]  [<ffffffff810d0b02>] handle_mm_fault+0x512/0x970
[ 7332.854776]  [<ffffffff813b068f>] do_page_fault+0x1bf/0x3e0
[ 7332.854781]  [<ffffffff813ae0af>] page_fault+0x1f/0x30
[ 7332.854833]  [<00007f75d16e2bbb>] 0x7f75d16e2bbb
[ 7332.854835] Mem-Info:
[ 7332.854837] Node 0 DMA per-cpu:
Removed CPU due to size limit in post
[ 7332.855884] active_anon:53475346 inactive_anon:87658 isolated_anon:0
[ 7332.855885]  active_file:4458 inactive_file:14191 isolated_file:0
[ 7332.855886]  unevictable:0 dirty:0 writeback:0 unstable:0
[ 7332.855887]  free:77055941 slab_reclaimable:4973 slab_unreclaimable:49548
[ 7332.855888]  mapped:28817 shmem:164169 pagetables:121624 bounce:0
[ 7332.855891] Node 0 DMA free:15540kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15256kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 7332.855902] lowmem_reserve[]: 0 1965 129225 129225
[ 7332.855905] Node 0 DMA32 free:153444kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2013036kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 7332.855915] lowmem_reserve[]: 0 0 127260 127260
***[ 7332.855918] Node 0 Normal free:19940kB min:16128kB low:20160kB high:24192kB*** active_anon:126081848kB inactive_anon:152652kB active_file:484kB inactive_file:784kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:130314240kB mlocked:0kB dirty:0kB writeback:0kB mapped:10172kB shmem:407496kB slab_reclaimable:5324kB slab_unreclaimable:120988kB kernel_stack:9328kB pagetables:253844kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:224 all_unreclaimable? no
[ 7332.855931] lowmem_reserve[]: 0 0 0 0
[ 7332.855934] Node 1 Normal free:103138276kB min:16384kB low:20480kB high:24576kB active_anon:28761824kB inactive_anon:68784kB active_file:7660kB inactive_file:34320kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:132382720kB mlocked:0kB dirty:0kB writeback:0kB mapped:66016kB shmem:89332kB slab_reclaimable:5780kB slab_unreclaimable:32864kB kernel_stack:4024kB pagetables:88136kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 7332.855946] lowmem_reserve[]: 0 0 0 0
[ 7332.855949] Node 2 Normal free:103206316kB min:16384kB low:20480kB high:24576kB active_anon:28808216kB inactive_anon:27560kB active_file:8556kB inactive_file:9868kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:132382720kB mlocked:0kB dirty:0kB writeback:0kB mapped:24012kB shmem:34824kB slab_reclaimable:4692kB slab_unreclaimable:22564kB kernel_stack:2352kB pagetables:71392kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 7332.855962] lowmem_reserve[]: 0 0 0 0
[ 7332.855965] Node 3 Normal free:101690248kB min:16384kB low:20480kB high:24576kB active_anon:30249496kB inactive_anon:101636kB active_file:1132kB inactive_file:11792kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:132382720kB mlocked:0kB dirty:0kB writeback:0kB mapped:15068kB shmem:125024kB slab_reclaimable:4096kB slab_unreclaimable:21776kB kernel_stack:1696kB pagetables:73124kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 7332.855977] lowmem_reserve[]: 0 0 0 0
[ 7332.855980] Node 0 DMA: 1*4kB 2*8kB 0*16kB 1*32kB 2*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15540kB
[ 7332.855989] Node 0 DMA32: 7*4kB 9*8kB 42*16kB 3*32kB 4*64kB 6*128kB 4*256kB 6*512kB 8*1024kB 4*2048kB 32*4096kB = 153444kB
[ 7332.855998] Node 0 Normal: 3041*4kB 5*8kB 2*16kB 6*32kB 26*64kB 5*128kB 1*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB = 20108kB
[ 7332.856006] Node 1 Normal: 565*4kB 334*8kB 191*16kB 161*32kB 4134*64kB 28130*128kB 1278*256kB 200*512kB 38*1024kB 12*2048kB 24113*4096kB = 103138260kB
[ 7332.856015] Node 2 Normal: 137*4kB 1036*8kB 541*16kB 297*32kB 4809*64kB 9018*128kB 732*256kB 358*512kB 164*1024kB 79*2048kB 24662*4096kB = 103205044kB
[ 7332.856025] Node 3 Normal: 4992*4kB 2165*8kB 571*16kB 232*32kB 178*64kB 550*128kB 232*256kB 68*512kB 19*1024kB 12*2048kB 24760*4096kB = 101690840kB
[ 7332.856034] 182818 total pagecache pages
[ 7332.856036] 0 pages in swap cache
[ 7332.856038] Swap cache stats: add 0, delete 0, find 0/0
[ 7332.856039] Free swap  = 0kB
[ 7332.856040] Total swap = 0kB
[ 7338.639674] 134217728 pages RAM
[ 7338.639677] 1861898 pages reserved
[ 7338.639678] 575892 pages shared
[ 7338.639679] 55216044 pages non-shared
[ 7338.639684] Out of memory: kill process 18141 (mpirun) score 1900895 or a child
share|improve this question

migrated from stackoverflow.com Nov 13 '12 at 2:33

This question came from our site for professional and enthusiast programmers.

    
It is more a sysadmin question than a programming one, so is off-topic here. You might want to use pmap on your process. And you could limit memory with ulimit shell builtin, or setrlimit(2) syscall. –  Basile Starynkevitch Nov 12 '12 at 20:40
    
I cannot limit the memory, I want this tool to take 95% of the available memory to exercise it. But for some reason malloc is not grabbing memory from the other nodes. If i do the code with numa_allocate_on_node it works as expected. But is not malloc supposed to give me memory from other nodes if one node runs out of memory ? –  dices Nov 12 '12 at 20:45
    
Thanks for the pmap command, it looks really useful for debugging –  dices Nov 12 '12 at 21:35

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.