The official answer to my question is that it can't be forced.
"you'll see that 8888 mentions that pages are displayed in IE8 Standards Mode (there's a typo though, it just says Internet Explorer 8!) regardless of the !DOCTYPE directive - however, it does not state that pages without a !DOCTYPE are displayed in Standards Mode."
So you have to have a doctype regardless.
Which is irrelevant either way, because selectors simply do not work in IE8 user style sheets applied via Accessibility options, regardless of doctype or document mode.