Be careful how you measure. In my experience, simply running
free -t will almost always show that all the Mem is used, and all the Swap is not used. That doesn't really give the whole picture.
Pay attention to the "
-/+ buffers/cache" line, which shows where the kernel is using memory for various stuff. This will show some of the "flex" available on the system. Don't plan on using all that cache space, though, as going to disk rather than that cache will also hurt performance.
You can probably effectively increase the memory in the box until the OS can't use all the memory for buffers/cache and see continued performance improvement.
Flipped around, and actually answering your question, don't let the buffers/cache get too small... actual numbers will vary on your system and with your load. Two samples on nearby machines show this cache between 700 MB and 1 GB, and 40-50% of total ram.
It's probably all tunable in the kernel somewhere, too.