Sorry for the even later answer, following up on @RedBullNinja's good comments.
Windows remembers default devices according to a timestamp representing to when they were marked as default.
Example 3 output devices: Speakers, headphones (3.5mm), and a Bluetooth headset.
You set your speakers as default 2 days ago, bluetooth headset yesterday, then finally 3.5mm headphones this morning. Whenever your 3.5mm headphones are connected (and enabled) that would get top dibs on preferred output, though once unplugged or disabled the bluetooth headset would now become the preferred output. If both the 3.5mm headphones and bluetooth headset are unplugged (or disabled) finally the speakers would be your output device.
Each time you click "Set Default" for a device the timestamp for that becomes most recent putting it at the top of the pile. You can therefore connect up all our possible output devices, enable them, then click "Set Default" for each one starting with your lowest preference to highest to define your exact order.
There are also two of theses lists: Once for regular audio, and again for communication audio. (You've probably already noticed the drop down caret on the default button to set default for just one type of audio there.)
What this doesn't answer is what happens with newly added devices never used before. I don't know if Windows would consider them to be the new top of the pile, and could be annoying for it to do so.
It's also annoying that you can't just see the current priority order without trawling through your system registry matching up somewhat random hardware IDs. (A small utility to see/manage that order would be very welcome!)