Either you're lucky to never have had corrupted data, or you're unlucky to never had noticed your data was corrupted.
When you perform an action that should write on a disk, most operating systems put the write operation into a queue. From time to time, they flush the queue. (I'm calling it a queue here, but actually operations can be performed out of order, operating systems do this when it's faster and gives the same final result.) This can make the write operations a lot faster, both because the system tries to perform them when it doesn't have anything better to do and because it can group them intelligently.
If you happen to unplug your device before everything has been written, you may miss the latest data. Worse, if the OS has been performing operations out of order, you may put your device into an inconsistent state and lose more than the latest data.
Some operating systems go into a more conservative (but slower) mode for removable devices, to reduce the risks associated with unplugging the device before it has been unmounted.
Doing operations out of order is sometimes not just a matter of speed. Cheap flash media (that doesn't to sector reallocation at the hardware level) has a limitation on the number of times you can write over any given sector. If you naively write all changes as they happen, this can kill the sectors that contain the file allocation table on a (V)FAT filesystem (the most common case for removable drives) or the journal on a typical modern filesystem. (See e.g. this discussion of
sync on the Linux Kernel mailing list.) Here, not updating the FAT or journal every time a file is written to is not just a big performance gain, it's also good for the lifetime of the hardware.
Until recently, Linux only gave a choice between
sync (write all changes as they happen) and
async (write whenever it's convenient). Recent versions introduce the
flush option for FAT filesystems, which is somewhere in between (flush all delayed writes as soon as the disk becomes inactive); it's on by default in Ubuntu 10.04.
On a different note, unmounting a removable drive ensures that no application has a file open. If you don't unmount before unplugging, you won't notice if you have unsaved data until it's too late. Unmounting while a file is open also increases the chance of corruption, both at the filesystem level (the OS may have queued some operations until the file is closed) and at the application level (e.g. if the application puts a lock file, it won't be removed).