You mentioned that you don't want to set up some other remote desktop solution but you care about performance. X11 is not a very efficient protocol, so you will only be able to expect minor improvements unless you use a more modern protocol.
NX (NoMachine) is probably your best choice. It still uses ssh, so it shouldn't be too much extra effort compared to other desktop protocols that might require changes to firewall settings, etc. There are NX packages for Fedora, so I presume they are also available for Ubuntu.
If you care enough about performance to take the time to ask the question, then hopefully you can spend a few more minutes to learn NX.
EDIT: To clarify why X11 over ssh will never be fast: the X protocol deals with low-level drawing like lines and circles, and with low-level events such as "the mouse moved 3 pixels to the left." Modern GUI toolkits like GTK and Qt don't draw lines, they draw images. When X11 goes over SSH, it must constantly send image data and low-level mouse events. A high-level protocol like NX, VNC, or Remote Desktop can reduce bandwidth and latency by being aware of how toolkits work. For example, they can avoid the need to send mouse events, they can avoid thousands of redraws when windows are moved, and they can cache areas of the screen like menus. If performance is even a minor concern, raw X11 is always the wrong choice. Fortunately, there are a multitude of fast alternatives that are easy to configure and use.
.ssh/configquote, no I did not. I've added
CompressionLevel 9to the host config. It seems a little faster now, but still not enough for comfortable use. Anything else I can do?
scpsays 1.2 MB/s when uploading or downloading to that remote machine, which more or less is the number that my ISP advertises.