I've got a problem with my debian server. It has 4GB of ram and 1GB swap. Swap is around 95% full and ram still has 80% left. I'm running lighttpd (max 40 users at the same time) and some gameservers (VALVe games) on it. Any ideas?
Does the system feel slow?
Maybe Linux sees that some of your servers are not in use and swaps them out to have more space for the running servers.
If you want more of your data in the ram, then you can try to change the value of Swapiness: http://en.wikipedia.org/wiki/Swappiness