The main risk is forgetting that you're running an ssh server and putting a weak password on an account. There are attackers out there that systematically try common account names (like webmaster and bob) and weak passwords. You can eliminate this risk by forbidding password logins (put PasswordAuthentication no in sshd_config, and either also put UsePAM No or disable password authentication in the PAM settings for ssh). An intermediate measure is to restrict ssh logins to a whitelist of users with AllowUsers or AllowGroups in sshd_config.
Note that allowing password logins is not in itself a security problem. Weak passwords and snooped passwords are the problems, and allowing password authentication in the ssh server is an enabler. To protect against password snooping, never type your password on a machine you don't fully trust (but then if you do trust a machine you might as well install a private key on it, and then you don't need password authentication).
The minimum requirement to use an ssh client on a machine is that you trust that there won't be an active hijacking of the ssh communication (a man-in-the-middle attack is possible if it's running on the client machine — you think you're typing commands into a pristine ssh client but the client is in fact transmitting your authentication data faithfully but also inserting a trojan horse into the communication afterwards). This is a weaker requirement than trusting there won't be a password snooper (usually performed through a keylogger, but there are other less automated methods such as shoulder surfing). If you do have the minimum trust but still fear snoopers, you can use one-time passwords (OpenSSH supports them through its PAM support).
Of course, as any other program that interacts with machines outside your control (not just network servers but also clients), you must keep up with security updates.