I should say from the start, I am aware that I could probably do with some more RAM as I'm currently running RStudio on Windows 10 with 4GB of RAM installed. Neither is the post necessarily solely related to R but memory handling at large. Freshly restarting the computer and RStudio, I typically have 2 to 2.5GB of 'available' RAM according to task manager.
Some of my code works flawlessly (particularly when I'm using data.table), even though it's doing quite a lot computationally; generating combinations and permutations, relatively complex joins. Other pieces of work will fail 4 out of 5 times with somewhat obscure, seemingly random at first, errors; e.g Value of SET_STRING_ELT() must be a 'CHARSXP'.
This isn't a code or file error or doing anything particularly complex code wise (just opening files, re-arranging some fields, mutating out of caps and writing them back out). If I rerun exactly the same piece of code a few times, or section by section, it'll eventually work with the only determining factor apparently being luck initially.
I identified some patterns in this. For example, it seems to be time related. If I manually drag over and run sections piece by piece, it'll work. And loops importing 10MB files using base R 'read.csv' will work along with rbindlist functions of larger files; right up to the 'available' RAM limit in task manager. But if I try looping through base R 'read.csv' type imports of 100MB files the error will start showing up, even when I explicitly remove the object from the environment, call gc() immediately afterwards, there is apparently >10x as much RAM available according to task manager, on a fresh restart and with absolutely nothing else running. The only solution I came up with for this was to add a 10 or more second system sleep after each gc() and 'read.csv' cycle; which is ridiculous when these files should take a few hundred milliseconds to read from the SSD (Kingston V300, ~500MB/s) but also mysteriously works (the Value of SET_STRING_ELT() must be a 'CHARSXP' errors disappear).
I was planning to do some upgrading of the computer anyway (buy more RAM) but I figured I'd do some investigating through performance monitor running some pieces of work to see what is actually bottle-necking the computer to begin with (if buying higher speed RAM is worth it etc); as the i3 4130t processor (one of Intel's cheapest) is rarely ever running above 50% with all four logicals apparently busy (using Microsoft MRAN R Open).
Looking at a differing piece of code, which loops through a 10 or so MB table of UID's and subsets a second table, and at the performance monitor results, I noticed there is a consistent climb in page faults as soon as I click run; it'll be up around 5000/s a minute or so in with the system cache continually dropping. Interestingly, this also seems to correspond with the loop gradually slowing down. It'll take a couple of minutes to cover 5% of the entries. But six or so hours later when I get back, it'll be half way through, crawling along, and any slight disturbance at all will cause R to fully hang. I also frequently have R reset itself or the entire OS; Windows has helpfully informed me, upon blue screening an hour or few into a run, that it's usually hit a page fault error.
There is a possibly related mention of something similar on the plotly forum:
I read an interesting and highly up voted post which I now can't find (but think was posted here) regarding the page file in which the user pointed out there is never actually any 'free' RAM in Windows; it's constantly filled preemptively, things are paged and then dumped out if something else needs the space.
There seem to be some extremely mixed opinions on whether or not to enable the page file.
I've tried both enabling and disabling it and see the same pattern in page faults occurring.
I seem to observing something similar to modidum on the plotly forum. Even though there is apparently more than enough 'available' RAM for these tasks, R seems to be trying to page file a whole lot of things.
I am curious if this could possibly be something to do with memory prioritisation within more recent versions of Windows. I'm aware that I can increase a process's priority in task manager, however does this actually increase it's memory allocation priority as opposed to just the processor thread priority? Is there anyway to permanently set such priorities without using proprietary software? I realise Windows is attempting to help by preemptively caching things in the RAM, however this doesn't actually seem to be helping at all with R. Is there anyway to selectively force or alter the caching profile? For more memory intensive work, I would prefer it if there was nothing cached that I'm not actually using.
For anyone who's curious about the SSD, despite doing fairly large numbers of read/writes to the page file and purposefully reading and writing to the drive from within R a lot (hundreds of thousands of files at a time, saturating it's capacity, then clearing it and re-saturating it over and over again), the SSD itself seems to be holding up fine; according to Kingston's diagnostic tool, there's basically nothing wrong with it even after years of use.
Thanks for clicking.
