Yes, you can limit the RAM the system will use in msconfig. (Boot tab, Advanced options button)
However, this is not really a viable way to troubleshoot this problem.
The first thing to do is to run MEMTEST86+. If it doesn't report any memory errors then the problem is very likely not with your RAM, and will not be solved by limiting Windows' use of RAM.
So... you've removed one of two DIMMs and the system is fine. The next step would be to take the DIMM you removed and install it in place of the remaining one.
If the problem comes back, you have a bad DIMM. Replace it.
If the problem doesn't come back, leave this DIMM where it is and put the other one in the other slot. (Result being that you've interchanged the DIMMs.) If the problem comes back, you have a problem :) because you have two DIMMs that test good in one arrangement but not in the other. this leads one to suspect the motherboard. It is also possible to have weak incompatibilities between RAM and motherboard, which will manifest themselves in this way. This usually happens with cheap mobos and/or cheap RAM.
Don't overlook the power supply either.
Note that "a particular level of RAM usage" does not really correspond to use of a particular range of RAM addresses. Suppose you have 8 GB RAM; that corresponds to addresses 0 through 0x1FFFFFFFF. Now suppose the system hangs after Task Manager reports "75% used". That does NOT mean that it's hanging when RAM at addresses 0x180000000 and up are being used; virtual memory OSs don't work that way. Even if only 25% is "used" per TM, that 25% that's "in use" is likely scattered all through the possible RAM addresses.
DIMMs do not usually correspond to specific ranges of addresses either. Most mobos that support two DIMMs are implementing "dual channel" RAM with interleaving. This means that the first 8 bytes of physical memory addresses will be in one DIMM, then the next 8 bytes in the other, etc., alternating back and forth.