I was having a similar problem recently and came to discover that an auto-refreshing AJAX page I was keeping loaded in a Firefox tab was causing Firefox to just start eating memory (like, 2 gigs of RAM on my 4 gig machine). From my perspective, it seemed locked up; it actually took a couple of minutes for a Ctrl-Alt-F1 to finally display the text console (at that point I was able to see the memory issue). And this is on a pretty fast quad core desktop.
So, is it possible that you have some program running in the background that is eating up memory while you are away, and that the system is just being really unresponsive?
Another thought; if you can ssh to the box you could try killing the Xscreensaver process.
toporps -efmagic on the command line to see what's going on? – Arjan Oct 2 '09 at 23:55