As long as your CentOS system is up-to-date on patches, i.e., you don't have a fairly old version of sshd running on the system, you should be fine. In regards to your comment that "there's not a horde of people just itching to get inside my network", if you check connection attempts to port 22 you might be surprised. If you run SSH on the default port of 22, you should expect that every day there will be people throughout the world trying to gain access to your system via SSH. I also have a home CentOS server and see thousands of attempts every day; about 14,000 attempts from 109 unique IP addresses in the last 12 hours. When I check /var/log/secure, most of the attempts are usually for a userid of root or admin, though I see attempted SSH logins for lots of other userids as well.
When I also ran an FTP server on the system, I would monitor the userids and passwords used via KRIPP and could see dictionary attacks every day where name dictionaries were used for userids and paired with English language dictionaries and commonly used passwords for passwords. Those occurred on a daily basis and I would observe as many as half a dozen systems in various parts of the world, e.g., Brazil, China, the U.S., etc., conducting such attacks at once.
You aren't allowing password logins, so don't need to worry about any accounts on the system having a weak password providing an entry point to the system for an attacker, but you still should be aware that there are thousands of systems out on the Internet scanning vast IP ranges searching for a vulnerable system. Sometimes they are looking for a system with financial or other information they can exploit and at other times it is for vulnerable systems that can be used as a launchpad to attack other systems while concealing the true origination point for whatever nefarious deeds the attacker has planned.