Actually, neither of these solutions worked, since my ability to uninstall via the Extension Manager was disabled and VisualSVN didn't appear in my list of programs through the Control Panel. Even after using Revo Uninstaller to reveal it in my list of Programs, uninstalling it failed.
After a bit more research, I found this thread. Essentially, you have to run VS2010 as administrator (right click application icon, then choose "Run as Administrator"). Only then will the Uninstall button appear in the Extension Manager so you can remove it.