Sometimes I have observed in Ubuntu's System Manager that without the physical RAM being used completely, the swap memory starts getting used.
Why is it so?
From the Ubuntu Wiki Swap Faq
I feel that the linux kernel does this optimally, and quite well, so you need not worry at all. You could reduce swappiness to increase your RAM usage, but I would not recommend that.
The OS sends pages to swap
I have no proof of this, but I would assume it does this when it has some spare time, so it can do it without impacting normal operations.