Can't ssh into a Linux server after ISP was changed

Question

The organization which network this server is a part of is changing its ISP. When I switch this server to new values of IP, gateway, etc., I can't connect to it via ssh anymore. I went to one of the port-checking tools and when I checked port 22 it returned "I could not see your service on _new_IP_value_ on port 22".

New IP is static. There's no router on the server in question, no port-forwarding -- the network cable goes straight into the server.

Naturally, I thought the organization's network or the new ISP was blocking port 22 so I phoned network's administrator and asked to open it or tell ISP to open it. He said he would. But nothing changed. I called him a day later with the same question, but you could tell he was busy -- said that he did what I asked and basically hung up on me.

So, now I'm thinking perhaps there's something wrong with the server's settings after all. But all the Linux tools such as lsof, netstat, nmap, etc. tell that sshd is up and running and listening on port 22, port 22 is open, etc. and I still can ssh into the server when I switch it back to our old ISP's network -- all I switch is IP, gateway, dns, nothing more.

Could there be something on this server that is blocking ssh connections when the server is on the new ISP's network, but not the old one's? The OS is Linux Mint.

Answer 1

Run a packet capture on the client, and run a packet capture on the server. The most common tools are Wireshark and tcpdump:

tcpdump -n -i eth0 "(tcp port 22 or 3389) or icmp or icmp6"

• If you see the TCP SYN packets leaving the client, but not arriving at the server, the problem is usually with the network – most likely at the server's ISP, because online port-checking tools have ruled out the possibility of client-side ISP being the odd one.

  (Maybe the admin didn't open the ports at all, or didn't open them for the correct address. It is not uncommon for port 3389 to be closed at ISP level too.)

  I think tcptraceroute can possibly help to discover how far the packets go before disappearing.

• If you see TCP SYN packets arriving at the server, but not getting responded to at all, the problem is with the server. It might be the server's internal firewall (tcpdump gets packets before they hit the firewall), so check whether you have any iptables rules and any nft rules.

  Also make sure the server has a route back to the client, and that it indeed goes to the correct gateway through the correct interface:

  ip -4 route get <client_ip>
  ip -4 route show match <client_ip>
  

  Also run systemctl cat sshd to check if there are any cgroup-level IP address whitelists. Also run ip xfrm policy to check if there are any IPsec policies.

• If you see TCP SYN packets arriving, but they generate TCP RST responses – it could still be the firewall, or it could be that the SSH daemon is configured to listen on the wrong address. (You didn't mention what local address is shown by lsof/netstat next to "port 22". If it shows 0.0.0.0:22 or [::]:22 that's good, but if it shows the old address, that must be changed in sshd_config.)

• If you see TCP SYN packets arriving, SYN/ACK responses coming out from the server, but the client never receives those responses – also a network problem, and again most likely on the server ISP side.