Can't connect publically access (forwarded) SSH on archlinux server

Question

I'm experiencing problems wanting to access my SSH server over my public IPv4 address, in my local network everything works fine though.

The connection would work as expected locally:

[alex@deathstar ~]$ ssh 192.168.178.86 -p 1338
Last login: Wed Jan  8 12:30:51 2020 from 192.168.178.85
[alex@bielefeld ~]$

But fatally fail publicly:

[alex@deathstar ~]$ ping 62.XXX.XXX.XXX
PING 62.XXX.XXX.XXX (62.XXX.XXX.XXX) 56(84) bytes of data.
64 bytes from 62.XXX.XXX.XXX: icmp_seq=1 ttl=63 time=13.3 ms
(...)
[alex@deathstar ~]$ ssh 62.XXX.XXX.XXX -p 1338
ssh: connect to host 62.XXX.XXX.XXX port 1338: Connection timed out

I'll give a quick summary:

• My server is using Arch Linux (2020.01.01)
• The server's firewall is disabled
• I've forwarded the port on my router (Fritz!Box 7590 version 07.12)
• Everything works fine inside my local network
• Pinging the public IPv4 address works

Also these checks I did may be useful:

Firewall settings:

I'm using Uncomplicated firewall.

[root@bielefeld /]# ufw status
Status: active

To                         Action      From
--                         ------      ----               
1338/tcp                   ALLOW       Anywhere                   
1338/tcp (v6)              ALLOW       Anywhere (v6) 

Network connections (Server):

You can also see my other computer being connected over the local network.

[root@bielefeld /]# netstat -tupan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:1338            0.0.0.0:*               LISTEN      4068/sshd           
tcp        0      0 192.168.178.86:1338     192.168.178.85:53868    ESTABLISHED 3326/sshd: alex     

Port scan locally (From server):

[root@bielefeld /]# nmap -p 1338 localhost
Starting Nmap 7.80 ( https://nmap.org ) at 2020-01-08 13:10 CET
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000043s latency).
Other addresses for localhost (not scanned): ::1

PORT     STATE SERVICE
1338/tcp open  SSH

Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds

Port scan on the server (From client)

[alex@deathstar ~]$ nmap -p 1338 62.XXX.XXX.XXX
Starting Nmap 7.80 ( https://nmap.org ) at 2020-01-08 13:26 CET
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.04 seconds

SSH daemon configuration (/etc/ssh/sshd_config)

Port 1338
ListenAddress 0.0.0.0

PermitRootLogin no
MaxAuthTries 3
MaxSessions 6

PubkeyAuthentication yes
AuthorizedKeysFile  .ssh/authorized_keys

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
PermitEmptyPasswords yes

UsePAM yes

AllowAgentForwarding yes
AllowTcpForwarding yes
GatewayPorts yes
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes
PermitTTY yes
PrintMotd no # pam does that

AllowUsers alex

Thank you so much in advance! I've grown deseperate to this point.

Answer 1

To establish the SSH connection from the internet to the server behind your home router, routing and nat must be configured correctly. Let's assume your client can successfully connect via SSH to other servers with a public IP:

1. Carrier-Grade NAT: Your ISP has to route incoming packets to your router. Verify that your ISP assigns the correct IP (62.x.y.z) to your router's WAN interface. Some ISPs will assign a private IP address to the customers' routers and employ CGN so that multiple customers are routed through the same public IP address.

This can be checked on the FritzBox web interface on the "Online" or "Internet" page. If the public ip is not assigned on the WAN interface, or another public IP is configured, then your ISP may not forward the port (1338) to your router or your router is misconfigured. Some ISPs assign more than 1 IP to customers.

2. Router NAT + Firewall: Your home router must handle the port forwarding (DNAT) correctly and must not block traffic through a firewall. The DNAT rule should forward TCP packets from anywhere destined for your public IP (62.x.y.z) on port 1338 to your home server (192.168.178.86) on port 1338. Additionally, the firewall must not block the traffic. Virtually all consumer home routers will create implicit firewall rules for forwarded ports, including the FritzBox. The router must also masquerade traffic from LAN to WAN, which again, virtually all home routers do without additional configuration.

Make sure the right protocol (TCP) is set in the port forwarding rule and the actual numeric port is set instead of a protocol name (SSH) using the protocol's default port. Also verify that the DNAT target is the correct destination LAN IP (192.168.178.86), as your router may use a cached DHCP lease that points to an incorrect IP if the server's IP was changed manually. The destination IP must be on the same subnet as the router, found in the advanced network settings on FritzBoxes. Make sure there are no rules blocking the outbound (reply) TCP traffic.

3. Server Firewall/NAT: Once the incoming TCP packet is routed to the server, it must not be dropped by any local firewalls or forwarded elsewhere by NATs in order to be delivered locally.

Check the default policy for INPUT and OUTPUT chains in iptables -L -v -n as well as NAT in iptables -L -v -n -t nat.

4. Server routes: Since the router applies Destination NAT, the original sender IP is left untouched in the TCP header. Therefore, the server will send replies to the original sender IP. A valid default route (gateway) must be configured on the server in order to send replies back through the same router where incoming TCP packets were forwarded from, or else the original sender may discard the server's reply due to it coming from a different IP than the original destination IP.

Check the configured routes on the server using ip route and make sure that there is a valid default route and no ambiguous routes using multiple network interfaces (ip rule). The default gateway must be within the subnet of the server, displayed by ip addr.

5. IPv4/IPv6: Make sure all of the above applies for both IPv4 AND IPv6 or disable one of them for testing. Some ISPs may only provide a public IPv6 subnet while the public IPv4 address is shared by many customers ("DS-Lite"), so IPv6 may be the only way to access the server successfully.

Routers may prefer one over the other, so incoming IPv4 could be sent via IPv6 internally or vice versa. You may need to delete the existing DHCP lease for the server to delete the preferred but defunct DNAT rule.

Lastly, do not test a connection to your router's WAN IP since the router is most likely not configured for hairpinning.