Boot from USB and mount encrypted partition
The only way that I can think of that will allow you to do that is to partition your drive and use an unencrypted partition to boot a Linux distro. That will then mount the encrypted partition after you type in the appropriate passphrase. Since the cleartext is only ever kept in memory, the data on the key itself is always protected, even if you pull it out mid-write. Of course, if you pull the drive mid-write, you'll corrupt data, but that has nothing to do with the encryption.
As far as portable apps go, you can install them to a third partition that is accessible to windows. Encrypting applications is unnecessary unless you're running some really exotic stuff (in which case you'd know how to do it anyway.) just make sure your apps don't leave fingerprints all over. Since a modern linux distro will run on pretty much any x86 machine, you'll also be able to access your data anywhere.
Another advantage from this is that you leave absolutely no marks on the host machine.
As a sidenote, because of flash memory wear, I advise doing full byte-for-byte backups of the entire key regularly, especially since you'll be running an os from it.
If you're paranoid (and I know you are) then you must not use the unencrypted OS if you have not kept a careful chain of custody, since some clever clog could have modified it to get your passphrase (via email, for example). If you misplace it for a few days, nuke the unencrypted partition and reinstall the os from scratch. (after a
dd if=/dev/random of=/dev/sdxn where xn is your OS partition ID). Also make sure that the OS partition is mounted as read-only, that way no program will inadvertently leave info in it.
This is fairly watertight, but not exactly simple to execute. It does, however, fulfill all of your requirements. Unfortunately, you'll have to boot from your usb key if you want access to your data.