This question ( Scroll shell output with mouse in tmux ) already answers the base question of how to mouse-scroll UP in tmux, but this method also automatically exits copy mode once it detects a scroll down past the current screen. According to this answer this is expected behavior too!
This is very disorienting as you're brought right back to the bottom of your prompt once you scroll down, and is not the behavior I'm looking for. Is there any way to disable this?