Ughh. Task Manager is a fairly poor tool to measure performance.
For example, when I evaluate how a sports team is doing, do I look at their statistics from one game, or many games?
On top of that, the human brain puts more emphasis, and thus weight, on recent data points, versus maybe more significant but distant data points.
The myth of free memory
From my understanding, the system should always keep a few "free" (i.e. zeroed) pages so it can >map those quickly when a process requests it (so pages can be cleared from another CPU while the >requesting process can proceed).
Zeroing memory pages is a fairly trivial process on today's machine. Disk access, even with top of the line SSDs, dominate the list of sluggish performance factors. There is a reason why Free memory and disk access is correlated in performance, since to an application, they are one and the same thing: secondary storage. While low memory can cause an OS to go to disk, from my observations of perfmon graphs, disk access and network latency (Ughh, I hate our ISP) dominate today's performance concerns.
It is also impossible to read the impact of having Free memory (zeroed pages) on your machine. We have no idea about the peak commits, peak # of page faults, # of pages, etc. I'd caution against jumping to conclusions over what is literally a snapshot.
My suggestions
Take many data points.
If you have local admin access to your machine, fire up perfmon
and use the default System Performance Data Set Collector. Have it collect data points every 10 to 15 seconds and run that for several days.
Also take a look at where in your workflow a faster machine would help out. It is possible that a faster machine may do very little. For example, VS2010 can be a memory hog during builds, as it has to load up all the dependencies of your project. I know a few places where they take several old Pentium 4s and "outsource" builds to them.
The "sluggishness" could also be attributed to the application themselves. Not all programs are designed to take advantage of lots of free memory. Some applications are rather aggressive about keeping a low memory profile, but rather keep data on disk since disk space is of little concern.