You'll want to use ulimit
ulimit can be used to limit memory utilization (among other things)
Here is an example of setting memory usage so low that
/bin/ls (which is larger than
/bin/cat) no longer works, but
/bin/cat still works.
$ ls -lh /bin/ls /bin/cat
-rwxr-xr-x 1 root root 25K May 24 2008 /bin/cat
-rwxr-xr-x 1 root root 88K May 24 2008 /bin/ls
$ date > test.txt
$ ulimit -d 10000 -m 10000 -v 10000
$ /bin/ls date.txt
/bin/ls: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Cannot allocate memory
$ /bin/cat date.txt
Thu Mar 26 11:51:16 PDT 2009
Note: If I set the limits to 1000 kilobytes, neither program works, because they load libraries, which increase their size. above 1000 KB.
-d data segment size
-m max memory size
-v virtual memory size
Run ulimit -a to see all the resource caps ulimits can set.