I am planning to keep the server running in text mode, but the client can ssh -X to the server and run GUI applications. How to start X server but not to the GUI desktop.
In this scenario, the server does not need X running. When you do
You can start X without starting a desktop on the machine you are running ssh from by issuing