0

I should say from the start, I am aware that I could probably do with some more RAM as I'm currently running RStudio on Windows 10 with 4GB of RAM installed. Neither is the post necessarily solely related to R but memory handling at large. Freshly restarting the computer and RStudio, I typically have 2 to 2.5GB of 'available' RAM according to task manager.

Some of my code works flawlessly (particularly when I'm using data.table), even though it's doing quite a lot computationally; generating combinations and permutations, relatively complex joins. Other pieces of work will fail 4 out of 5 times with somewhat obscure, seemingly random at first, errors; e.g Value of SET_STRING_ELT() must be a 'CHARSXP'.

This isn't a code or file error or doing anything particularly complex code wise (just opening files, re-arranging some fields, mutating out of caps and writing them back out). If I rerun exactly the same piece of code a few times, or section by section, it'll eventually work with the only determining factor apparently being luck initially.

I identified some patterns in this. For example, it seems to be time related. If I manually drag over and run sections piece by piece, it'll work. And loops importing 10MB files using base R 'read.csv' will work along with rbindlist functions of larger files; right up to the 'available' RAM limit in task manager. But if I try looping through base R 'read.csv' type imports of 100MB files the error will start showing up, even when I explicitly remove the object from the environment, call gc() immediately afterwards, there is apparently >10x as much RAM available according to task manager, on a fresh restart and with absolutely nothing else running. The only solution I came up with for this was to add a 10 or more second system sleep after each gc() and 'read.csv' cycle; which is ridiculous when these files should take a few hundred milliseconds to read from the SSD (Kingston V300, ~500MB/s) but also mysteriously works (the Value of SET_STRING_ELT() must be a 'CHARSXP' errors disappear).

I was planning to do some upgrading of the computer anyway (buy more RAM) but I figured I'd do some investigating through performance monitor running some pieces of work to see what is actually bottle-necking the computer to begin with (if buying higher speed RAM is worth it etc); as the i3 4130t processor (one of Intel's cheapest) is rarely ever running above 50% with all four logicals apparently busy (using Microsoft MRAN R Open).

Looking at a differing piece of code, which loops through a 10 or so MB table of UID's and subsets a second table, and at the performance monitor results, I noticed there is a consistent climb in page faults as soon as I click run; it'll be up around 5000/s a minute or so in with the system cache continually dropping. Interestingly, this also seems to correspond with the loop gradually slowing down. It'll take a couple of minutes to cover 5% of the entries. But six or so hours later when I get back, it'll be half way through, crawling along, and any slight disturbance at all will cause R to fully hang. I also frequently have R reset itself or the entire OS; Windows has helpfully informed me, upon blue screening an hour or few into a run, that it's usually hit a page fault error.

There is a possibly related mention of something similar on the plotly forum:

"It seems like it is constantly reading/writing to the page file. After about 5 minutes, the CPU usage is at 20% and the number of page faults is ~15,000,000. After 10 minutes, 30% CPU usage and 65,000,000 page faults."

I read an interesting and highly up voted post which I now can't find (but think was posted here) regarding the page file in which the user pointed out there is never actually any 'free' RAM in Windows; it's constantly filled preemptively, things are paged and then dumped out if something else needs the space.

There seem to be some extremely mixed opinions on whether or not to enable the page file.

I've tried both enabling and disabling it and see the same pattern in page faults occurring.

I seem to observing something similar to modidum on the plotly forum. Even though there is apparently more than enough 'available' RAM for these tasks, R seems to be trying to page file a whole lot of things.

I am curious if this could possibly be something to do with memory prioritisation within more recent versions of Windows. I'm aware that I can increase a process's priority in task manager, however does this actually increase it's memory allocation priority as opposed to just the processor thread priority? Is there anyway to permanently set such priorities without using proprietary software? I realise Windows is attempting to help by preemptively caching things in the RAM, however this doesn't actually seem to be helping at all with R. Is there anyway to selectively force or alter the caching profile? For more memory intensive work, I would prefer it if there was nothing cached that I'm not actually using.

For anyone who's curious about the SSD, despite doing fairly large numbers of read/writes to the page file and purposefully reading and writing to the drive from within R a lot (hundreds of thousands of files at a time, saturating it's capacity, then clearing it and re-saturating it over and over again), the SSD itself seems to be holding up fine; according to Kingston's diagnostic tool, there's basically nothing wrong with it even after years of use.

Thanks for clicking.

2
  • 1
    The large majority (often huge majority) of page faults have nothing to do with the pagefile. Typically they are soft page faults which are resolved within memory with no disk access involved. I don't know why but some applications tend to produce a lot of them.
    – LMiller7
    Jun 14 '17 at 17:59
  • 1
    Rather than looking elsewhere I would look into the programs code. There are all kinds of reasons why faulty code will exhibit intermittent problems.
    – LMiller7
    Jun 14 '17 at 18:07
1

Looking at a differing piece of code, which loops through a 10 or so MB table of UID's and subsets a second table, and at the performance monitor results, I noticed there is a consistent climb in page faults as soon as I click run; it'll be up around 5000/s a minute or so in with the system cache continually dropping. Interestingly, this also seems to correspond with the loop gradually slowing down.

I'm not a developer so can't help you with the first part of your question sorry, but as an Engineer I can shed some light on the Hardware/OS relationship.

Please understand there is no easy way to explain this without first deep diving into the fundamental differences (and similarities) of an Operating System and the Platform Hardware. But here goes:

Also it would help you to know that at it's most basic level the entire Platform is one long staircase of cache levels, either physical cache for the CPU ( L1, L2, L3, L4, RAM, HDD and so on), or Virtual cache levels for Processes and OS Memory Mangler. (Process Private Working Set, Working Set, Standby etc.).

Page Faults come in two flavors, Soft and Hard. A Soft Page Fault happens when a process requests a page which is not located within it's working set, AKA the range of addresses available to the process. The page is usually sitting in RAM as part of the "standby" list in Task Manager (cached files).

The description Standby is misleading because in truth all pages mapped by the CPU are part of the system working set. Even cached files.

A CPU knows the location of a requested page in primary (RAM) or secondary (HDD) storage - RAM or the HDD (AKA cache levels - u see?). It doesn't care about anything else.

The CPU doesn't move page, it moves the pointer.

So in summary: A Soft Fault is paging in and out of the process address space - eg Working Set to Standby and back again. It's not really a big issue though.

A Hard Page Fault happens when the requested page is not in RAM but on the Hard Drive page file. Hard Page Faults don't occur when the Page File is turned off (obviously).

If Soft Faults when there is Free Memory can reduced by increasing the working set size (registry and GPO Editor), adding RAM, or both.

I read an interesting and highly up voted post which I now can't find (but think was posted here) regarding the page file in which the user pointed out there is never actually any 'free' RAM in Windows; it's constantly filled preemptively, things are paged and then dumped out if something else needs the space.

Not true.

There should always be at least some Free RAM for best performance so pages can be read straight in in from the drive. Otherwise standby pages first have to be discarded and page tables updated. This takes time.

If there isn't any Free then the machine needs more RAM.

There seem to be some extremely mixed opinions on whether or not to enable the page file.

I've tried both enabling and disabling it and see the same pattern in page faults occurring.

The Page File was a requirement for Intel IA-32e/Intel-64 Processors with 32bit address pins to RAM running Windows x86 with PAE or Windows 64.

The page file was the only way these CPU's could reach addresses over 4GB which the the OS was fully capable of doing.

Contrary to popular myth, PAE in the OS stands for Page Address Extension NOT Physical Address Extension. Page Address Extension allows addresses higher than 4GB to be reached by the OS, provided the CPU has 36bit internal registers.

If Page Address Extension is enabled on CPU's with 32bit registers, all hell breaks loose. 32/32 CPU's (32 External pins/ 32 Internal registers) can reach addresses up to 4GB.

Edit: I've added some more random but relevant points to help clarify the big picture......hopefully I haven't overdone it.

E**Note: earlier I incorrectly named IA-64 as x86-64, it should have read Intel-64.

IA-64 is x64.

** 32/36 (IA32e/Intel 64) AKA x86-64 can address higher than 4GB 2x 4GB segments. One 4GB segment is RAM, the other segment is the pagefile. Primary and secondary storage. RAM ---> CPU: external address pins, CPU------> HDD: internal data registers.

36bit Page Address Extension reduces per process address space on IA32e/Intel64 down to 3.5GB, 512MB is reserved for the CPU page table directory and the extra 4bits are used for the segment directory pointer

Ever wonder why x87 compiled games never seem to use more than 3.6GB? It's because high pointers are truncated by the Intel compiler. The other ~512MB is marked as reserved. On 64bit hardware the process around 500MB of VAD's is permanently marked free space.

Intel IA-32e/Intel-64 is otherwise known as x86-64. x86-64: a CPU with 32 pins to RAM capable of pages 4GB by way of internal registers and a page file on the Hard Drive.

None of the above affects RAM btw, a CPU with 32 pins cannot communicate with memory modules over 4GB in density. This is a hardware limitation.
It would be like trying to call your friend from a landline with no line - just a phone. :P


Physical Address Extension is an Intel CPU architecture nomenclature referring to the CPU address pins to RAM. The above is clearly stated in Intel docs.

The page file has never ever been required on CPU's with 36bit address pins. (AMD64/IA64)

BTW, related articles such as found on Wikipedia, Technet, MSDN etc pertaining to Windows memory limits and PAE are completely wrong or misleading for the most part.

Microsoft are the worst offenders in this regard.

I am curious if this could possibly be something to do with memory prioritisation within more recent versions of Windows. I'm aware that I can increase a process's priority in task manager, however does this actually increase it's memory allocation priority as opposed to just the processor thread priority? Is there anyway to permanently set such priorities without using proprietary software? I realise Windows is attempting to help by preemptively caching things in the RAM, however this doesn't actually seem to be helping at all with R. Is there anyway to selectively force or alter the caching profile? For more memory intensive work, I would prefer it if there was nothing cached that I'm not actually using.

It should be caching as much as possible

Great article here which debunks a lot of misinformation: File Cache Performance and Tuning.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.