We have also experienced this issue on Windows Server 2008 R2 in our company. In our case the machine got totally stuck. It seems to be a bug on this server version of Windows, because it did not occur under the same circumstances either on my much smaller Windows 7 machine running in VirtualBox, or under Linux.
You can set the cache to a certain limit with an API as others have already posted, but you can find applications to directly use it, such as SetSystemFileCacheSize.exe or NT Cache Setter.
The downside is that you need to set the cache after every reboot. You can also flush the whole cache in one go with this.
We set the file cache on our VM having 14 GB RAM to 4 GB and it works. Also flush works nicely and you can see it directly in Task Manager.
Command line samples:
Show the current Windows settings:
Set the minimum size to 8 MB and the maximum size to 128 MB:
SetSystemFileCacheSize.exe 8 128
Disable minimum size and set the maximum size to 128 MB:
SetSystemFileCacheSize.exe off 128
Disable minimum and maximum size:
SetSystemFileCacheSize.exe off off
Flush the file cache:
Note: The tool must be started with admin privileges.