To bring a vocabulary point : what you seek is a "persistent" Live USB, or persistent USB installation.
Knowing that term you can readily find instructions from a lot of web pages.
You have a million option. Distros specialized in live CDs may be nice (Knoppix, Puppy). Historically they ran from CD and allowed to saved data on floppy, HDD or USB (still can do), working purely from USB is a natural evolution for them.
Ubuntu is nice as I presume you know it well. Linux Mint is the same, but has flash, java, codecs out of the box even on a DVD. Choose a distro variant with a lightweight desktop if needed.
For rescue (command line only, boot to six root terminals) PLD Rescue CD is very likable (lots of stuff like dd, sshfs, ntfs-progs, chntpw, fdisk etc.)
For Windows I may mention there's Windows 8 "ready to go" but you need the Enterprise edition and this costs money.