In Windows 7 go to "System Properties".
In advanced tab click "Settings" of "Performance" section.
Here under "Processor Scheduling" select "Adjust for .." "Background services".
This is only based on personal experience and I can't confirm or quote exact reference into MSDN or some other published source. BTW, just clarifying that my personal experience is fairly significant as I have managed many servers and workstations over the years.
My theory is that by adjusting for "background services", windows keeps the background apps (including minimized ones) at a decent priority and hence lessen their chance of getting swapped out.
One more from very "vague" memory of reading about swap file.. Even the EXE's get swapped out, i.e. if a process is not actively using portions of its executable code, then that portion may be swapped out (not not really written to swap file). In other words, RAM used to hold code (as opposed to data) is also freed up and used for other processes. Later when you switch to the app, and it runs a portion of its code that is no longer in memory, Windows has to then reload it as needed. This causes a lag as well. I believe setting for background services also reduces this effect.
Secondly, since you don't use more than 2.5 G of memory, you could make the swap file as minimal as windows will allow without pain (leave enough for it to be able to write small memory dump in case of BSOD).
You can completely remove the pagefile but I don't advise that because as I mentioned above, windows will complain about not being able to write a crashdump.