User switching requires additional resources for keeping the user accounts active.
In order to speed it up, you would need more RAM at least. To further increase performance a faster hard drive would help to retrieve data from the page file. As well as a faster CPU.
These "upgrades" are very generic and would speed up your entire system.
Alternatively you could disable windows services that you do not require (Indexing, ReadyBoost, Hibernation, SuperFetch) that run per user.
To see what services are running per user simply open task manager and select services.
You can sort by Status and Group to filter which services are in use and which don't belong to any groups.
To control which services and applications run at start up or log on which would use system resources you can do the following
WindowsKey+R -> Type in "msconfig"
You can manage all startup applications, registry entries and services from here.
Using Fast User Switching in Windows 7