I have a little linux box (running debian) connected to a monitor that I want to put at some public location to display some stuff. I can SSH into it and do not want anyone to be able to access it by plugging in a keyboard or mouse. However when I start x via the command line, that is exactly what people can do. How can I prevent that?
Well, to display one X application full-screen without anyone logging in, the most convenient way to go is to use a specialized display manager such as
That is all is there is to it if the machine is physically secured, and the screen is the only thing general public has access to.
If you will have the whole box exposed, then things obviously change, and you have decide for yourself what level of interaction you need.
I would say that if you have no way to physically secure the box, then all bets are off anyway: if anyone is free to plug a dongle in your box then sooner or later they will do something "interesting" with your machine no matter whether you block external ports/blacklist drivers etc.
On the other hand, if the box is supposed to be in a controlled environment, say, the system case itself is concealed and only the screen, keyboard and pointing device are exposed then such a system is colloquially called "a kiosk", and to set it up properly your task is to prevent the user from interfering with the system using its input devices.
The minimal check list of the things to secure would be this:
Note that there are specialized distros to run browser-based kiosks, and HOWTOs on how to set up a typical general-purpose distro to run a kiosk — just google using these keywords.