I can see questions asking how to prevent Windows Update from (asking for) restarting the computer, but I don't see one asking how to prevent it from keeping a backup for potential uninstall later. I know it is possible to do this manually via commmand line option if one download & install the update outside Windows Update, so I assume it's possible to set some registry to prevent it for Windows Update as well?
Windows XP
IT would be a safer way to let the uninstall information be saved for starters and then once the install is successful remove the uninstall information.
This blog post has a script for such cleanups -- Remove Windows Update Uninstall Information.
Tested on Windows XP, Requires WScript and VBScript (which would be with most Windows OS)
You don't say which Windows version you are talking about (if you are asking about a specific version that is).
Under Vista, Windows update uses System Restore Points - it creates one before installing updates (even, rather annoyingly, Windows Defender pattern updates).
You can prevent the Windows Installer from making System Restore Points by setting (or creating) the registry key HKLM\SOFTWARE\Policies\Microsoft\Windows\Installer\LimitSystemRestoreCheckpointing with a value of 1.
A neater way to do this on Business and Ultimate is to run gpedit.msc and in Computer Configuration, Administrative Templates, Windows Components, Windows Installer, double-click the "Turn off creation of System Restore Checkpoints" and set this to Enabled.
Be aware through, that this would disable System Restore Point creation for ALL software installations, not just Windows Updates.
I don't know of any way to change Windows Update behaviour.
