I am having the same problem as you. I agree that files (setup.exe for example) are being overwritten which is causing the problem. I have tried to extract Windows 8 files from the ISO to a folder in the root of the USB drive called "WIN8" and manually create a boot entry that boots WIN8/bootmgr but when I boot it up it still defaults to Windows 7 install. I think it is looking for setup.exe and other files at the root of the USB and there seems to be no way to change that.
Your solution of making a separate partition seem to be the best workaround for the issue. The problem is that removable media (like a USB memory stick) cannot have more than 1 partition because Windows recognizes these devices as removable media and (for some reason) will only read the first partition. However, if you use a portable USB HDD (like a WD My Passport) you can initialize it with an MBR partition table (don't use the GUID partition Table) and add up to 4 primary partitions (or 3 primary partitions and a bunch of logical partitions). Only primary partitions can be bootable.
You can use YUMI to add system tools, Linux distros, etc and Windows 7 on the first primary partition and copy the contents of your Win8 DVD/ISO to the second partition.
Now manually add this entry to multiboot\menu\menu.lst on the partition where you installed your Linux distros and Windows 7 with YUMI.
title Install Windows 8
root (hd0,2)
chainloader /bootmgr
And then Windows 8 installer should boot from the second partition.
I would love to see a solution where it can be done on a large flash drive instead of a portable hdd if only for the convenience since portable/external hard drives are not really pocket size (well.. not like a flash drive).
