2

I try to use a large chunk of memory in R, of a size of about 7.6 GB. My system has 16 GB RAM, so I did not expect this to be a problem. However, R prevents this, and attempting to circumvent it leads to massive crashes of R and various other applications (web browsers). The system reported bus problems, but I do not have exact error messages since the system eventually crashed.

My question is: What happened? How can I prevent this and allocate more memory in R (or any application)?

I have the feeling that it may be related to how much memory is addressable, not the memory that is theoretically available.

Details

I attempted to use a bigger chunk of memory in R, a matrix with 1bn entries, about 7.6 GB. R does not easily allow vectors/matrices of that size, though it is not clear to me, why. (It results in Error: cannot allocate vector of size 7.6 Gb) However, R has libraries such as bigmemory that are supposedly able to deal with large vectors. From the R interpreter:

> library(bigmemory)
Loading required package: bigmemory.sri
> bx <- big.matrix(45070,45070)

 *** caught bus error ***
address 0x7ff75ffac000, cause 'non-existent physical address'

Traceback:
 1: .Call("bigmemory_CreateSharedMatrix", PACKAGE = "bigmemory",     row, col, colnames, rownames, typeLength, ini, separated)
 2: CreateSharedMatrix(as.double(nrow), as.double(ncol), as.character(colnames),     as.character(rownames), as.integer(typeVal), as.double(init),     as.logical(separated))
 3: big.matrix(45070, 45070)

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 

So R crashed, but it can be saved by choosing 2 and cancelling the exit. It may not have been very smart to try the same again, but anyway, here we go:

Selection: 2
Save workspace image? [y/n/c]: c
> bx <- big.matrix(45070,45070)
terminate called after throwing an instance of 'boost::interprocess::interprocess_exception'
  what():  No space left on device
Aborted (core dumped)

From the journal log, it looks like this:

Aug 23 14:49:25 system systemd-coredump[426]: Process 423 (R) of user 1000 dumped core.

                                           Stack trace of thread 423:
                                           #0  0x00007ff94bab18c0 raise (libc.so.6)
                                           #1  0x00007ff94bab2f72 abort (libc.so.6)
                                           #2  0x00007ff94774d035 _ZN9__gnu_cxx27__verbose_terminate_handlerEv (libstdc++.so.6)
                                           #3  0x00007ff94774ac46 _ZN10__cxxabiv111__terminateEPFvvE (libstdc++.so.6)
                                           #4  0x00007ff947749b49 __cxa_call_terminate (libstdc++.so.6)
                                           #5  0x00007ff94774a538 __gxx_personality_v0 (libstdc++.so.6)
                                           #6  0x00007ff9474b3ee3 _Unwind_RaiseException_Phase2 (libgcc_s.so.1)
                                           #7  0x00007ff9474b470e _Unwind_Resume (libgcc_s.so.1)
                                           #8  0x00007ff945279da6 _ZN21SharedMemoryBigMatrix7destroyEv (bigmemory.so)
                                           #9  0x00007ff9452a7762 _Z15CreateRAMMatrixI21SharedMemoryBigMatrixEP7SEXPRECS2_S2_S2_S2_S2_S2_S2_ (bigmemory.so)
                                           #10 0x00007ff94528d79c bigmemory_CreateSharedMatrix (bigmemory.so)
                                           #11 0x00007ff94c11a33a n/a (libR.so)
                                           #12 0x00007ff94c11a8c6 n/a (libR.so)
                                           #13 0x00007ff94c158fb8 Rf_eval (libR.so)
                                           #14 0x00007ff94c15ba3b n/a (libR.so)
                                           #15 0x00007ff94c158d5b Rf_eval (libR.so)
                                           #16 0x00007ff94c15adce n/a (libR.so)
                                           #17 0x00007ff94c150963 n/a (libR.so)
                                           #18 0x00007ff94c158938 Rf_eval (libR.so)
                                           #19 0x00007ff94c15adce n/a (libR.so)
                                           #20 0x00007ff94c158b02 Rf_eval (libR.so)
                                           #21 0x00007ff94c15cbc7 n/a (libR.so)
                                           #22 0x00007ff94c158d5b Rf_eval (libR.so)
                                           #23 0x00007ff94c181f92 Rf_ReplIteration (libR.so)
                                           #24 0x00007ff94c1823b1 n/a (libR.so)
                                           #25 0x00007ff94c182468 run_Rmainloop (libR.so)
                                           #26 0x000000000040074b main (R)
                                           #27 0x00007ff94ba9e4ca __libc_start_main (libc.so.6)
                                           #28 0x000000000040078a _start (R)
-- Subject: Process 423 (R) dumped core
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: man:core(5)
-- 
-- Process 423 (R) crashed and dumped core.
-- 
-- This usually indicates a programming error in the crashing program and
-- should be reported to its vendor as a bug.

System-wide consequences

At this time, neither a desktop environment nor any graphical applications were running. I started the window manager and a browser to look up what was happending. To my horror, I found that neither Firefox, nor Opera or Chromium would start. The error messages said something about bus errors, but I do not have exact error messages since the system eventually crashed. It is notable that other applications, even bigger ones, such as libreoffice, could be started without problems. Could it be that this has something to do with addresses needed to establish network connections? Could it be that the system was somehow out of addresses after R crashed? (I do not understand, however, why that would persist after the R process died.)

From the journal log, it looks like this (long stack traces truncated):

Aug 23 15:16:19 system systemd-coredump[18050]: Process 18017 (firefox) of user 1000 dumped core.

                                             Stack trace of thread 18017:
                                             #0  0x00007ff72e679018 sem_init@@GLIBC_2.2.5 (libpthread.so.0)
                                    (...)

Aug 23 15:16:20 system systemd-coredump[18097]: Process 18062 (firefox) of user 1000 dumped core.

                                             Stack trace of thread 18062:
                                             #0  0x00007f2098a98018 sem_init@@GLIBC_2.2.5 (libpthread.so.0)
                                    (...)

Aug 23 15:16:21 system systemd-coredump[18144]: Process 18109 (firefox) of user 1000 dumped core.

                                             Stack trace of thread 18109:
                                             #0  0x00007f2d45410018 sem_init@@GLIBC_2.2.5 (libpthread.so.0)
                                    (...)
                                    (...)
                                    (...)

Aug 23 15:19:16 system systemd-coredump[19510]: Process 19370 (opera) of user 1000 dumped core.

                                             Stack trace of thread 19395:
                                             #0  0x0000000001c882f7 n/a (opera)
                                             #1  0x0000000001c890e9 n/a (opera)
                                    (...)

Aug 23 15:20:58 system systemd-coredump[20140]: Process 20136 (evas_image_load) of user 1000 dumped core.

                                             Stack trace of thread 20136:
                                             #0  0x00007fba4432babd __memset_avx2_erms (libc.so.6)
                                    (...)

Aug 23 15:30:11 system systemd-coredump[20990]: Process 20958 (WebKitWebProces) of user 1000 dumped core.

                                             Stack trace of thread 20958:
                                             #0  0x00007fc5dd5ed7d0 n/a (libpixman-1.so.0)
                                             #1  0x00007fc5dd5d273b n/a (libpixman-1.so.0)
                                    (...)

Aug 23 15:31:07 system systemd-coredump[22406]: Process 20936 (midori) of user 1000 dumped core.

                                             Stack trace of thread 22403:
                                             #0  0x00007f3a38d5b6df __memmove_avx_unaligned_erms (libc.so.6)
                                             #1  0x00007f3a39759e78 n/a (libwebkit2gtk-4.0.so.37)

I then attempted to restart dbus (which was also not the smartest move and crashed the system).

Other aspects

Before the system crashed, I also realized the following:

[user@system ~]$ df -h
Filesystem         Size  Used Avail Use% Mounted on
dev                7.6G     0  7.6G   0% /dev
run                7.6G  788K  7.6G   1% /run
/dev/mapper/root   412G   89G  324G  22% /
tmpfs              7.6G  7.6G     0 100% /dev/shm
tmpfs              7.6G     0  7.6G   0% /sys/fs/cgroup
/dev/sda1          2.0G   52M  1.8G   3% /boot
tmpfs              7.6G     0  7.6G   0% /tmp
tmpfs              1.6G     0  1.6G   0% /run/user/1000
[user@system ~]$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15469         146        7438        7735        7884        7349
Swap:         14335           0       14335
[user@system ~]$ 

Why are the virtual filesystems (dev, run, tmpfs) all of size 7.6GB, exactly what R would not allocate?

I have verified that it is possible to allocate as much as 6.7GB in R, but somewhere below 7.6GB there is a limit. No maximum memory is set in R or in the system:

[user@system ~]$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 61833
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 61833
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

... and in the R interpreter:

> Sys.getenv("R_MAX_MEM_SIZE")
[1] ""
> Sys.getenv()
COLUMNS                 235
DBUS_SESSION_BUS_ADDRESS
                        unix:path=/run/user/1000/bus
DESKTOP                 Enlightenment
DISPLAY                 :0.0
E_BIN_DIR               /usr/bin
E_CONF_PROFILE          standard
E_DATA_DIR              /usr/share/enlightenment
E_ICON_THEME            gnome
E_IPC_SOCKET            /run/user/1000/e-user@0/633
E_LIB_DIR               /usr/lib
E_LOCALE_DIR            /usr/share/locale
E_PREFIX                /usr
E_RESTART               1
E_SCALE                 1.000
E_START                 enlightenment_start
E_START_TIME            1503499246.8
E_TAINTED               NO
EDITOR                  vi
HOME                    /home/user
LANG                    en_GB.UTF-8
LD_LIBRARY_PATH         /usr/lib64/R/lib:/usr/lib/jvm/java-7-openjdk/jre/lib/amd64/server
LINES                   58
LN_S                    ln -s
LOGNAME                 user
M2                      /opt/maven//bin
M2_HOME                 /opt/maven/
MAIL                    /var/spool/mail/user
MAKE                    make
MAVEN_OPTS              -Xmx512m
MOZ_PLUGIN_PATH         /usr/lib/mozilla/plugins
PAGER                   /usr/bin/less
PANTS                   ON
PATH                    /opt/maven//bin:/home/user/Applications/.bin:/usr/bin:/opt/maven//bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
PWD                     /home/user
QT_QPA_PLATFORMTHEME    gtk2
QT_STYLE_OVERRIDE       gtk2
R_ARCH                  
R_BROWSER               /usr/bin/xdg-open
R_BZIPCMD               /usr/bin/bzip2
R_DOC_DIR               /usr/share/doc/R/
R_GZIPCMD               /usr/bin/gzip
R_HOME                  /usr/lib64/R
R_INCLUDE_DIR           /usr/include/R/
R_LIBS_SITE             
R_LIBS_USER             ~/R/x86_64-pc-linux-gnu-library/3.4
R_PAPERSIZE             a4
R_PDFVIEWER             /usr/bin/xdg-open
R_PLATFORM              x86_64-pc-linux-gnu
R_PRINTCMD              
R_RD4PDF                times,inconsolata,hyper
R_SESSION_TMPDIR        /tmp/RtmpXBvepb
R_SHARE_DIR             /usr/share/R/
R_SYSTEM_ABI            linux,gcc,gxx,gfortran,?
R_TEXI2DVICMD           /usr/bin/texi2dvi
R_UNZIPCMD              /usr/bin/unzip
R_ZIPCMD                /usr/bin/zip
SED                     /usr/bin/sed
SHELL                   /bin/bash
SHLVL                   3
TAR                     /usr/bin/tar
TERM                    xterm
USER                    user
WINDOWPATH              1
XAUTHORITY              /home/user/.Xauthority
XDG_CONFIG_DIRS         /usr/etc/xdg:/etc/xdg
XDG_DATA_DIRS           /usr/share/enlightenment:/usr/share:/usr/local/share:/usr/share
XDG_MENU_PREFIX         e-
XDG_RUNTIME_DIR         /run/user/1000
XDG_SEAT                seat0
E_PREFIX                /usr
E_RESTART               1
E_SCALE                 1.000
E_START                 enlightenment_start
E_START_TIME            1503499246.8
E_TAINTED               NO
EDITOR                  vi
HOME                    /home/user
LANG                    en_GB.UTF-8
LD_LIBRARY_PATH         /usr/lib64/R/lib:/usr/lib/jvm/java-7-openjdk/jre/lib/amd64/server
LINES                   58
LN_S                    ln -s
LOGNAME                 user
M2                      /opt/maven//bin
M2_HOME                 /opt/maven/
MAIL                    /var/spool/mail/user
MAKE                    make
MAVEN_OPTS              -Xmx512m
MOZ_PLUGIN_PATH         /usr/lib/mozilla/plugins
PAGER                   /usr/bin/less
PANTS                   ON
PATH                    /opt/maven//bin:/home/user/Applications/.bin:/usr/bin:/opt/maven//bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
PWD                     /home/user
QT_QPA_PLATFORMTHEME    gtk2
QT_STYLE_OVERRIDE       gtk2
R_ARCH                  
R_BROWSER               /usr/bin/xdg-open
R_BZIPCMD               /usr/bin/bzip2
R_DOC_DIR               /usr/share/doc/R/
R_GZIPCMD               /usr/bin/gzip
R_HOME                  /usr/lib64/R
R_INCLUDE_DIR           /usr/include/R/
R_LIBS_SITE             
R_LIBS_USER             ~/R/x86_64-pc-linux-gnu-library/3.4
R_PAPERSIZE             a4
R_PDFVIEWER             /usr/bin/xdg-open
R_PLATFORM              x86_64-pc-linux-gnu
R_PRINTCMD              
R_RD4PDF                times,inconsolata,hyper
R_SESSION_TMPDIR        /tmp/RtmpXBvepb
R_SHARE_DIR             /usr/share/R/
R_SYSTEM_ABI            linux,gcc,gxx,gfortran,?
R_TEXI2DVICMD           /usr/bin/texi2dvi
R_UNZIPCMD              /usr/bin/unzip
R_ZIPCMD                /usr/bin/zip
SED                     /usr/bin/sed
SHELL                   /bin/bash
SHLVL                   3
TAR                     /usr/bin/tar
TERM                    xterm
USER                    user
WINDOWPATH              1
XAUTHORITY              /home/user/.Xauthority
XDG_CONFIG_DIRS         /usr/etc/xdg:/etc/xdg
XDG_DATA_DIRS           /usr/share/enlightenment:/usr/share:/usr/local/share:/usr/share
XDG_MENU_PREFIX         e-
XDG_RUNTIME_DIR         /run/user/1000
XDG_SEAT                seat0
XDG_SESSION_ID          c1
XDG_VTNR                1
XMODIFIERS              @im=ibus

Software

R version is 3.4.1; the system is Arch Linux.

[user@system ~]$ uname -a
Linux system 4.11.9-1-ARCH #1 SMP PREEMPT Wed Jul 5 18:23:08 CEST 2017 x86_64 GNU/Linux
3

This is one of the very few occasions on which I disagree with grawity, whose answers are always enlightening, to many and also certainly to me.

The reason is that the filling up of /dev/shm is not caused by some other process, so that it can be easily freed for use by the R package, but instead it is caused by the big.memory module inside R itself! So freeing /dev/shm is tantamount to killing R.

The bigmemory package manual states, on page 1:

Description: Create, store, access, and manipulate massive matrices. Matrices are allocated to shared memory and may use memory-mapped files.

This clarifies an important point: you cannot expect to use all of your memory by using big.memory, only the portion allocated to /dev/shm, which is typically half of the total memory you have available. If you want to enlarge or shrink shm, modify the relevant line in /etc/fstab and reboot.

We can safely assume that the filling up of /dev/shm is due to R. In fact, the OP post clearly states that there were no other programs running at the time,

At this time, neither a desktop environment nor any graphical applications were running.

, so it is difficult to imagine what else (i.e., apart from R) might be swallowing up such a large chunk of shared memory.

In fact, it is also easy to understand the root of the problem. First of all, your matrix

bx <- big.matrix(45070,45070)

has 45070 x 45070 > 2 billions elements. Second, as per the R manual,

R has no single precision data type. All real numbers are stored in double precision format

and then

All R platforms are required to work with values conforming to the IEC 60559 (also known as IEEE 754) standard.

....

In IEEE 754-2008/IEC60559:2011 this is called ‘binary64’ format.

and the Wikipedia article on the binary64 format clearly states:

Double-precision floating-point format is a computer number format that occupies 8 bytes (64 bits) in computer memory.

Thus your more than 2 billion elements, each occupying 8 bytes, would like to occupy more than 16 billions bytes of memory, which is roughly twice as much as your /dev/shm (where big.memory would like to store them, see above), has available. Hence the crash, and the error message:

terminate called after throwing an instance of 'boost::interprocess::interprocess_exception' what(): No space left on device

This error message, from the Boost C++ libraries, pertains to a class of functions which:

Boost.Interprocess offers some basic classes to create shared memory objects and file mappings and map those mappable classes to the process' address space.

As for your system failing after the R core dump, it is well explained by grawity, in that /dev/shm has not been cleaned, and all processes which use shared memory (like everything using dynamic libraries, for instance) will fail due to the lack of space on the device: your easiest option is to reboot.

What are your options? First, perhaps you can install 32GiB of memory, which would brute-force your current predicament. Or, you may see whether your matrix really requires so many elements: for instance, symmetric matrices contain just a tad more than half of the elements of a non-symmetric matrix, and you would just need to enlarge /dev/shm a little bit. Or perhaps you are dealing with a sparse matrix, which would be even easier to compress than a symmetric matrix.

In other words, you will have to look at some of the details of the matrix and find a solution tailored to your concrete situation.

  • What are "bus processes"? – grawity Aug 24 '17 at 8:28
  • Ah I see. But... Even though libraries are shared in memory, that has completely nothing to do with /dev/shm – the kernel directly uses paging for that. The /dev/shm folder is purely for the POSIX "shm" IPC functions. – grawity Aug 24 '17 at 18:31
  • Ah I see. But... Even though libraries are shared in memory, that has completely nothing to do with /dev/shm – the kernel directly uses paging for that. The /dev/shm folder is purely for the POSIX "shm" IPC functions. – grawity Aug 24 '17 at 18:31
3

tmpfs filesystems grow on demand, so the "total size" you see is just the capacity limit – unless specified otherwise at mount time, the default limit equals 50% of physical memory. (This doesn't mean tmpfs is locked into physical memory; it can be swapped out.)

However, notice that one filesystem, /dev/shm, is reporting 7.6 GB used (i.e. filled to its limit). This location is where SHM segments (shared memory – an inter-process communication feature) are kept, although sometimes programs create miscellaneous files there directly as well.

SHM segments are persistent; if a program exits without explicitly removing them, they'll stay around. So if your previous runs kept setting up SHM and then crashing, that could easily fill half your RAM and leave only ~8 GB to new programs.

(And vice versa, since the default capacity of /dev/shm is 50% of physical memory, the total size of all SHM segments is limited to 7.6 GB. I doubt that's relevant here – I would be very surprised if a program legitimately needed a SHM segment that large.)

To clean up /dev/shm, you could a) reboot, or b) carefully remove files found there using plain old rm. But first always use lsof to make sure they're not in-use.

Alternatively, raise the size limit using:

mount -o remount,size=90% /dev/shm

(As a side note, you're running a rather old kernel for Arch Linux – the current version is 4.12.8, unless you run linux-lts.)

  • By “grow on demand” he means all that the /run filesystem as listed in the question only requires 788 KB (+ some overhead) of memory. tmpfs instances mostly come for free in terms of anything, really. – Daniel B Aug 23 '17 at 17:14
  • I believe this is not fully correct: the bigmemory package actually does use shared memory for the handling of the large matrices and this is what fills it up to the brink, and ultimately causes the crash. Pls read my reply below. – MariusMatutiae Aug 24 '17 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.