I find it useful to have a handy shortcut icon for the hardware removal operation. How do I create a shortcut for the Safely Remove Hardware dialog?
RunDll32.exe shell32.dll,Control_RunDLL hotplug.dll
Source: Create a Shortcut or Hotkey for the Safely Remove Hardware Dialog
chopeen's answer, if you find your USB device does not accept a remove operation.
Read these tips for pendrives.
You might also want to keep the RemoveDrive tool handy.