Does it really make sense to Explorer to exhaust 16GB of RAM to deal with 35,000 files in the Recycle Bin? I do have folders with that many files, and they don't cause any trouble.
Recycle Bin is different, because it maintains additional data, which needs to get parsed and presented in custom views:
A hidden file called info2 (info in Windows 95 without the Windows Desktop Update) stores the file's original path and original name in binary format. Since Windows Vista, the "meta" information of each file is saved as $I. and the original file is renamed to $R..
When a file is sent to the Vista Recycle Bin, it is renamed with a pseudorandom filename beginning with $R and ending in the file's original extension; this $R file contains the file's original content. Along with the $R file, another file (beginning with $I and named to be complimentary with the $R file) is created and contains the file's date and time of deletion (file offset 16; 8 bytes) and the file's path at the time of deletion (file offset 24; variable length; in Unicode) as shown in Figure 5.41.
Opening my Recycle Bin containing ~ 30'000 files currently consumes ~ 5,5 GB of RAM for Windows Explorer as well. While not as much as in your case, I have fewer files and things might depend on where files have been deleted from, influencing lengths of paths shown, consuming different memory and stuff.
So in my opinion seeing that much memory consumed in case of so many files is normal for Recycle Bin.
Shouldn't the OS have a mechanism to deny RAM to a program (even if it is an Explorer folder, which I have set up to launch on a separate process) instead of letting it crash the system?
It's difficult to decide 1. if the process consuming so much memory is doing the correct thing for the user at all and 2. how that process will behave in future. Remember that Windows Explorer doesn't start telling Windows to consume all memory available up until a point it gets unstable, but asked for more and more memory over time up until a point Windows recognized that it has a problem now. At that point it notified you as the user and asked for what to do, because you are the only one who can decide.
There are different strategies of course, the OOM-killer of Linux is one. That is simply a piece of software kicking in the exact same case like for Windows, but instead of asking some user, by default that killer decides itself. Sometimes, and in your case most likely, this means that the process consuming all that memory will get killed, optionally rescuing the system that way. But that's not necessarily the case: Think of situations were you are running VMs or databases consuming lots of memory and some new process starting to consume additional, but less than the former mentioned up until a point the system runs out of resources again. Depending on the current killer-settings/implementation/... it might simply be that instead of the new process one of your large VM- or database-processes gets killed, because the killer can't know which of the processes is the most useful one to you. It only sees which processes consume a lot of memory and knows that memory is needed.
Additionally keep in mind that preserving a functional state of the OS is only one aspect of the problem as well: The OS is a slave serving you as its master in the end, trying as hard as possible to process the workloads you put at it. Up until a point were it can't anymore, but simply because by executing processes you told it to do so.