The Linux kernel Control Groups is what you are looking for. With cgroups, you can limit the amount of resources certain processes can use, including memory. So in your case, you would create at least 2 cgroups. One would limit the memory access to all processes on the system to maybe 90% of your total RAM. Then the second one would have access to all the RAM. You would put your terminal process in the second cgroup. There are plenty of resources on the web that cover how to configure your cgroups and it depends somewhat on your particular distribution and kernel version.
Another, simpler option is to just disable swap on your machine. The "unresponsiveness" that you observe is your OS using the swap space. When you disable it, the oom_killer will terminate the process that is hogging all the memory so the system does not appear to hang.