Free memory is memory which has not been allocated for any purpose yet. Available memory is memory which was recently used by an application, and is likely to be used again, but it can be given up temporarily if needed (think, you minimized an application. Part of that application's memory can be reused until you restore the application and continue using it. This is what causes the long delay when restoring a minimized application on RAM-starved systems). Cached memory is memory which the system is using as a system cache for files which have been recently fetched from the harddrive.
As a general rule, windows will move stuff to the page file as RAM fills up, with increasingly severe performance degredation until the page file fills up, which is when it usually displays the low system memory at about 100-200mb of free memory left (or for me, anyways).
You might take a look at your "commit size" with the tool Process Explorer by Sysinternals (More or less Task Manager on steroids). This will tell you how much of the total available virtual memory is in active use. If the peak is close to the limit, then you might consider increasing the size of your pagefile or installing more memory.
Are you running a 64-bit copy of windows or a 32-bit copy of windows with a high-end graphics cards (ones with 512mb or more of video memory)? Because of the way addressing works, 32-bit windows can only "see" 4GB of memory, and this includes video memory. So if you have a graphics card with 512mb of memory, windows only really has 3.5gb instead of your total 4021mb to play with. If this is the case, you might consider switching to a 64-bit OS.