Unable to SSH from WAN into LAN device via real IP when OpenVPN is running

Question

I have an Ubuntu device with OpenVPN enabled and connected on the LAN behind a TP-Link router. Here is what is known:

• While inside the LAN I can SSH into the device directly (VPN started or stopped).
• When outside the LAN and OpenVPN is stopped, I can connect to the device via port forwarding (WAN port 22106).
• When outside the LAN and OpenVPN is started, I cannot SSH into the device via the real IP anymore.
• There is nothing special in iptables -L, just UFW noise

Any ideas how to SSH into the device via the real ISP IP, not the VPN IP, when the VPN connection is established?

Solution:

Here is one full solution that worked. Create a virtual adapter, assign it an unused neighboring IP, and add a rule and lookup table to use the same interface for the outbound packets. Also, forward the WAN port to device port 22 at that new static IP.

ip addr add 192.168.0.6 dev eth0:0
ip rule add from 192.168.0.6 table 1234
ip route add default via 192.168.0.1 dev eth0:0 table 1234

---

Previous attempts:

Here are all my failed attempts from various solutions around Stack Exchange.

---

cat /etc/iproute2/rt_tables
echo 200 isp2 >> /etc/iproute2/rt_tables


## FAILED - Direct ISP IP SSH okay, LAN SSH blocked
ip rule add from 192.168.0.106 table isp2
ip route add default via 192.168.0.1 dev eth0 table isp2


## FAILED - Direct ISP IP SSH okay, LAN SSH blocked
ip rule add from 192.168.0.106 table isp2 priority 900
ip rule add from dev eth0 table isp2 priority 1000
ip route add default via 192.168.0.1 dev eth0 table isp2
ip rule add iif eth0 table isp2 priority 1000


## FAILED - No effect
iptables -A PREROUTING -t mangle -p tcp --dport 222 -j MARK --set-mark 1
ip rule add fwmark 1 table isp2
ip route add default via 192.168.0.1 dev eth0 table isp2


## FAILED - No effect
iptables -t mangle -A OUTPUT -p tcp --dport 222 -o eth0 -j MARK --set-mark 1
ip rule add fwmark 1 table isp2
ip route add 0.0.0.0/0 table isp2 dev eth0 


## FAILED - No effect
ip rule add from 192.168.0.106 table isp2
ip route add table isp2 to 192.168.0.106/32 dev eth0
ip route add table isp2 default via 192.168.0.1


## FAILED - No effect
iptables -t mangle -A PREROUTING -i eth0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 1234
iptables -t mangle -A OUTPUT -m connmark --mark 1234 -j MARK --set-mark 4321
ip route add default dev eth0 table 3412
ip rule add fwmark 4321 table 3412


## FAILED - Direct ISP IP SSH okay, LAN SSH blocked
iptables -t mangle -A PREROUTING -i eth0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 1234
iptables -t mangle -A OUTPUT -m connmark --mark 1234 -j MARK --set-mark 4321
ip route add default via 192.168.0.1 dev eth0 table 3412
ip rule add fwmark 4321 table 3412


## FAILED - Direct ISP IP SSH okay, LAN SSH blocked
iptables -t mangle -A PREROUTING -i eth0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 1234
iptables -t mangle -A OUTPUT -m connmark --mark 1234 -j MARK --set-mark 4321
ip route add default via 192.168.0.1 dev eth0 table isp2
ip rule add fwmark 4321 table isp2

Update: More failed attempts

## FAILED
iptables -t mangle -A OUTPUT -p tcp --dport 22106 -o eth0 -j MARK --set-mark 1 && \
iptables -t mangle -A OUTPUT -p tcp --dport 222 -o eth0 -j MARK --set-mark 1 && \
iptables -t mangle -A OUTPUT -p tcp --dport 22 -o eth0 -j MARK --set-mark 1 && \
ip rule add fwmark 1 table isp2 && \
ip route add default via 192.168.0.1 dev eth0 table isp2

---

Update: Added tcpdump output of exchange between the sshd (192.168.0.106:22) and a LAN machine (192.168.0.114). The ACK packets are being sent back on eth0, but not reaching the .114 machine. At this point I CAN ssh from outside the LAN, but inside the LAN fails. The only ip rules added are:

# WAN SSH okay, LAN SSH fails
ip rule add from 192.168.0.106 table isp2
ip route add default via 192.168.0.1 dev eth0 table isp2

---

WAN (static IP) <--> Router <--> Server (192.168.0.106)

Attempted router forwarding:

22106/TCP --> 192.168.0.106:22 (same result as above)
22106/TCP --> 192.168.0.106:22106 (same result as above)

Device ip a when OpenVPN started:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:1e:06:36:2b:05 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.106/24 brd 192.168.0.255 scope global dynamic noprefixroute eth0
       valid_lft 3068726852sec preferred_lft 3068726852sec
    inet6 fe80::a15b:2d6f:2dc9:8892/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/none
    inet 10.7.0.10 peer 10.7.0.9/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::7d84:a8cc:f96e:6f11/64 scope link stable-privacy
       valid_lft forever preferred_lft forever

Device /etc/ssh/sshd_config (all else are defaults and commented out):

Port 22
Port 22106
ListenAddress 192.168.0.106:22
ListenAddress 0.0.0.0:22106
AllowAgentForwarding yes
AllowTcpForwarding all
GatewayPorts yes
X11Forwarding yes
PermitTunnel yes
DisableForwarding no

Device netstat -tlpn:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      455/systemd-resolve
tcp        0      0 0.0.0.0:22106           0.0.0.0:*               LISTEN      9312/sshd
tcp        0      0 192.168.0.106:22        0.0.0.0:*               LISTEN      9312/sshd

Device ufw status:

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
22106/tcp                  ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)
22106/tcp (v6)             ALLOW       Anywhere (v6)

Answer 1

to restate your issue differently:

I can connect to my server running OpenVPN, locally. I can connect to the same server from outside my network because i setup port forwarding However, any attempts to connect from outside my network will fail if OpenVPN is running because the default route for non-local traffic is down the tunnel, and there is no session that originated locally so it is dropped. I would like to connect to the server from outside my network, while my vpn is running.

I would suggest a virtual interface.

ifconfig eth0:0 192.168.0.107 (if this ip is free on your LAN)

will spin up a virtual subinterface on eth0. add port forwarding rules to use this IP and test. You should now have a management IP globally regardless of tunnel state.

Don't forget to persist it with proper interface config for your flavor of linux.

Answer 2

This is to be expected. When openvpn is running it.sets up a default route via the vpn, so outbound packets will go through that, and presumably be dropped due ti egress filtering run by most providers.

There are a number of solutions/workarrounds, none of.which are both easy and robust.

1. SSH to the OpenVPN IP, assuming of-course that this is a real IP or port mapped.

2. Use policy based routing on the ssh server to route ssh traffic out via the ethernet interface

2a. Use a specific (ie to external client IP) route out the router interface - more specific routes.take precendence so this can work but no traffic to that IP will go.vua VPN then.

3. add a VPN SERVER to the router and have the client connect to the VPN before.using SSH. Be sure.to advertise the LAN range over the VPN.

Update

I suspect your second and third failed rules were most of the way there, except that they used port 222 rather then port 22 or port 22106 and got some syntax wrong. I would try:

iptables -t mangle -A OUTPUT -p tcp --dport 222106 -o eth0 -j MARK --set-mark 1
ip rule add fwmark 1 table isp2
ip route add default via 192.168.0.1 dev eth0 table isp2