I think if you still can SSH to the box after putting bad passphrase to the SSH Key, mean that you doing something wrong there.
Passphrase of private key is used to access the key it self, and if you type bad passphrase then no action its taken with that SSH Key.
Please verify which keys you have loaded
ssh-add -l
and you should list all loaded keys to ssh agent.
And then after this you could have look into authorised_keys
file under ~/.ssh/authorised_keys
where you storing your publics keys which are used for private key verification when you connecting to remote server. And just remove equivalent public key which was generated to allow the private key to make connection.