It needs to have video, audio, mouse, keyboard and network connectors and to support nothing more than connecting remotely to various Windows PCs that will have all the real resources needed. Sort of a "pure terminal" hardware device. Is there such a thing? Or do I need to look at media boxes? (But I think those typically come with Android, not Windows, these days.) Or will it always be cheaper to just get the oldest PC that can handle the desired video bandwidth and be done with it?
I'd suggest you use a Raspberry Pi. But you will need some knowlegde for installing software on it. There are enough tutorials on the internet on how to install for example Windows 10(or 8).
If you want a relatively cheap device I'd suggest you a Thin Client from HP. They come with Windows preinstalled and you can use them directly. They do cost way more than a Raspberry Pi (Raspberry is around 30 dollar and a Thin Client around 300).
edit: Video&Audio is never a good idea over RDP, it is very slow and you will have alot quality losses. It seems you want some sort of mediastation? Why not use an old computer without the RDP?
Android or Ipad tablets can run remote desktop sessions in most situations (several free and paid RDP apps are available on either platform) and you can either use on-screen keyboard/mouse or get an external keyboard (for ipad or android) and external mouse for Android (either Bluetooth or usb-otg if supported). If you need a desktop-like system that can act as an RDP client, they exist as well. The Dell Wyse 5212 is an example of a purpose-built client-only desktop appliance like you are describing. You could use an Android "media client" box with keyboard, video, mouse to do the same, but it's not purpose built so user experience might suffer.