It won't use all of your memory, but you won't miss very much. 32-bit Windows is generally limited to 4GB of total memory address space. This address space is used for all memory-mapped devices in your system, including RAM, GPU RAM, DMA hard drives (hard drive cache), audio device memory, network card direct access, and numerous other small I/O devices. So what you have is 3GB RAM + 1GB GPU + various sundry low-level system components.
Those additional system components don't add up to a lot, but they will eat a small portion of your available address space. Exactly how much depends on your system, but it's generally less than 100MB all told. Compared to the bigger GPU and normal RAM you'll hardly notice it, but it will mean some of your physical ram goes unused.