PasswordAuthentication no, is there any security concerned reason to restrict my SSH port to a specific IP? Can I open my 22 port to the internet?
If you don't allow passwords for SSH logins, but, instead, restrict logins to key-based logins, you prevent malefactors from breaking into your system by guessing a password. If you run an SSH server that is accessible to the Internet at large on port 22, you will likely find that there are people trying to break into it on a daily basis via dictionary attacks, i.e, by attempting to log in with an account name likely to be present on the system, e.g., "root" or even a common name like "john", paired against every word in a dictionary as a password. It doesn't take an automated script long to try every possible word in a dictionary with a given user name, if there's no mechanism on the SSH server, such as Fail2ban to restrict access to the server from IP addresses from which many invalid login attempts have been observed.
But, restricting SSH logins to key-based authentication doesn't make you invulnerable to possible compromise. E.g., vulnerabilities have been found in previous versions of SSH. E.g., suppose a new zero-day vulnerability is found in the SSH protocol, if you were also restricting SSH logins to only specific IP addresses, you would have an additional level of protection, i.e., you would have a defense in depth strategy where you had another layer of protection outside of the mechanisms employed by the SSH server software. Sometimes, though, it isn't practical to know every IP address from which a legitimate SSH login needs to take place or even possible to limit access by the system's firewall to a range of IP addresses, but, if you can do so, that will provide you an additional level of protection.