My laptop is Acer Aspire 472 on Windows 8.1 and my USB drive is Pny 64gb. I want to install Ubuntu on my USB drive. So I can use it without installing it on my laptop. After installing it, I will have to press F2 or F12 button for going to the boot menu and select my USB drive to boot. But how can I come back to the regular Windows ?
Your computer starts by running a built-in program called BIOS. This is what you're seeing (and interacting with) when you press F2 or F12, and this is the program that gives you the option to override the normal boot device.
The Live version of Ubuntu is perfect for the situation you described in your question. Insert your properly formatted Live USB drive in your computer, and reboot. If you override the normal boot behavior of the computer (by pressing F12 when prompted), you can choose to boot to the Live CD operating system (i.e. Ubuntu).
When you choose to return to your Windows operating system, shut down Ubuntu, and reboot. This time, don't press F12. Allow the computer to boot normally. It will start Windows.
You can go back and forth between Windows and Ubuntu Live as often as you like.