One way to solve this is to configure prompt on your computer(s) so that it looks different, in a way specific to that machine. For example, you can assign a color to each machine and use that color for the host part of the prompt. (How to do that is another question.)
When you start an ssh session, your local terminal will be receiving the color codes along with prompt, so (as long as your prompt is configured correctly) you do not have to do anything special to revert it back.
Then using any terminal emulator that supports colors should result in correct coloring.
I use this approach quite frequently with PuTTY and several remote machines (all Debian). Even
sshing from one session to other (i.e., connecting via PuTTY to machine A and issuing
ssh B) works perfectly.
However, the pitfall is that not all terminals will support colors correctly, but from your question I can't see how big deal is it for you.