I have always scraped it off with a credit card, then used alcohol applied with paper towels and a q-tip to clean the residue.
Wikipedia has more detail though:
Computer processor heatsinks utilize a variety of designs to promote better thermal transfer between components. Some thermal greases have a durability up to at least 8 years. Flat and smooth surfaces may use a small line method to apply material, and exposed heat-pipe surfaces will be best prepared with multiple lines.
Excess grease separating the metal surfaces more than the minimum necessary to exclude air gaps will only degrade conductivity, increasing the risk of overheating. Silver-based thermal grease can also be either slightly electrically conductive or capacitive; if some flows onto the circuits it can cause malfunctioning and damage.
Over time, some thermal greases may dry out, have reduced heat transferring capabilities, or set like glue and make it difficult to remove the heat sink. If too much force is applied the processor may be damaged. Heating the grease by turning the processor on for a short period often softens the adhesion. Another method to use can be by turning the heatsink slowly instead of lifting it up. It is recommended that thermal grease be re-applied with each removal of the heatsink.
Silicone oil-based thermal grease can be removed from a component or heatsink with an alcohol (such as rubbing alcohol) or acetone. Special-purpose cleaners are made for removing heatsink grease and cleaning the surfaces.