If your ssh client is new enough, you can use the config file
Match keyword to have conditional configuration based on the target port.
Match host somehost exec "test %p = 42"
The above example would only apply the "IdentityFile" line if the target host is "somehost" and the target port is 42.
Match works like
Host in that the lines following it (up to the next Match or Host line) only apply to sessions which match the criteria. In the example, the Match line checks that the host is "somehost", and it also runs an external command which is deemed to match if the command exits with status 0.
The command being run here is
test %p = 42. "%p" will be replaced by the port value that ssh would use up to that point (either the default of 22 or the value from the command line).
test is also known as
[; it's a command-line utility mostly used in shell scripts as part of an
if statement. Here,
test is being used to test if the port number is 42.
Match ssh_config keyword was added to OpenSSH in version 6.5 or 6.6. Check your
ssh_config man page to see if it's available on your system.