Basically The Reboot would depend on the Changes made by the software. To answer your questions:
What is the computer doing technically that requires a reboot?
If the Software is installed on windows normally reboot would fix the registry changes of the software.
Why can't it update or install programs without rebooting?
Let me say your installation software creates one new user account, it cant be done without affecting/interrupting the actions of kernel. That is the changes made in software would interrupt the basic properties of os. In order to smoothen the process the software vendors force you to reboot.
Why hasn't Microsoft or others figured out how to update/install programs without reboots?
Its their core architecture. You might noticed the linux machines wont need reboot since:
From booby's answer:
The major reason for this behavior is that Linux doesn't lock executed
files and libraries, which allows direct replacement of those files
and does only require the applications to restart. For installations
is the reason the package-management-systems, while in Windows every
program installs all needed libraries (even if they're already
installed, but when they are in use they are locked, which needs a
restart to clear the situation) in Linux an application only
references the needed packages which are installed once (and never
again), reducing the overhead.