What is special about Windows RT that makes it unable to run regular Windows programs that didn't come from the app store?
Microsoft has locked down Windows RT.
- Windows RT devices implement Secure Boot and do not allow disabling it or installing different keys
- Windows RT only runs signed binaries, so no unapproved third-party code can run.
The above does not have anything to do with ARM - there's nothing stopping Microsoft from making a non-ARM device that behaves like the above. Windows has supported a configuration where only signed binaries can be run for a long time, and newer systems can be locked down with Secure Boot. So it's nothing too new, though on RT systems you can't disable these options.
Microsoft did the above to sort of provide an appliance-like experience like Apple does with the iPad--e.g. programs are vetted, approved, sandboxed, and the operating system cannot be modified.
I understand that Windows RT was used with ARM processors, but some of my old Windows 98 programs ran fine on an Intel Pentium and still run fine on more modern AMD processors
AMD has always made CPUs that are compatible with Intel - specifically with Intel's "x86" architecture. Even then, AMD and Intel chips implement certain newer features differently--different enough where the program needs to know what type of CPU it's running on and change what it does a little, but not different enough where most programs won't run.
ARM is a completely different type of CPU that was a new design (from a UK company called "Acorn RISC Machines") and it has a totally different architecture. It's incompatible with Intel.