Non-NAT gateway through OpenVPN

Question

I have several servers (B, C, ...) which are in one place and one machine (A), which is behind a regular old home router. I want all of them to be able to connect to each other. It seems to me that OpenVPN is a great place to start here, so I set up a tunnel (tun0) between A and B over the internet. This is what the situation looks like:

 10.8.0.4                 10.8.0.1
----------  10.8.0.0/24   ---------- 
|        |                |        | 
|   A   tun0 ----------- tun0  B   | 10.128.140.204 
|        |                |        | 
----------                -- eth1 -- 
                              |
                              | 10.128.0.0/16
                              |
                          -- eth1 --
                          |        |
                          |    C   | 10.128.13.224
                          |        |
                          ----------

Specifically, I need A to be able to reach C under the address 10.128.13.224 and C to be able to reach A under the address 10.8.0.4. This means that NAT won't do - most resources talk about setting up NAT on B so A can see C - and this is where it seems to get tricky.

So this is what I've been trying to do now:

• tell A to use 10.8.0.1 as gateway for 10.128.0.0/16
• tell B to use 10.128.140.204 as gateway for 10.8.0.0/24
• get B to act as that gateway

Routing table on A:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         [home router]   0.0.0.0         UG    0      0        0 enp0s3
10.8.0.0        *               255.255.255.0   U     0      0        0 tun0
10.128.0.0      localhost       255.255.0.0     UG    0      0        0 tun0
[home ip]       *               255.255.255.0   U     0      0        0 enp0s3

A# ip route get 10.128.13.224
10.128.13.224 via 10.8.0.1 dev tun0  src 10.8.0.4
    cache

Looks great. Routing table on C:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    0      0        0 eth0
10.8.0.0        10.128.140.204  255.255.255.0   UG    0      0        0 eth1
10.13.0.0       *               255.255.0.0     U     0      0        0 eth0
10.128.0.0      *               255.255.0.0     U     0      0        0 eth1
[public IP]     *               255.255.255.0   U     0      0        0 eth0

C# ip route get 10.8.0.4
10.8.0.4 via 10.128.140.204 dev eth1  src 10.128.13.224
    cache

Great. Enable IP forwarding on B:

B# cat /proc/sys/net/ipv4/ip_forward
1

I guess the default ACCEPT everything rules in iptables would also work, but for good measure, I limited for FORWARD chain a bit:

B# iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-A FORWARD -s 10.128.0.0/16 -d 10.8.0.0/24 -i eth1 -o tun0 -j ACCEPT
-A FORWARD -s 10.8.0.0/24 -d 10.128.0.0/16 -i tun0 -o eth1 -j ACCEPT

Well, they don't. A# ping 10.128.41.180 does nothing and neither does C# ping 10.8.0.4.

IF I add NAT as follows:

B# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -s 10.8.0.0/24 -d 10.128.0.0/16 -o eth1 -j MASQUERADE

then a ping from A to C works.

A# ping 10.128.13.224
PING 10.128.13.224 (10.128.13.224) 56(84) bytes of data.
64 bytes from 10.128.13.224: icmp_seq=1 ttl=63 time=120 ms
...

This is not what I want to do, but I guess it means that something is working. Another curious thing is that if I configure the MASQUERADE the other way round, a ping from C to A does not work, so I guess there could be a clue here.

Now I'm starting to wonder if what I want to do even can be done or if I have some basic misunderstanding of how networking works. Is this even possible? Am I overlooking something basic? I'd appreciate any hints or resources.

Answer 1

Turns out that the above will work beautifully in a regular setting. The servers (B, C, ...) however were in a network which doesn't allow sending packets which have a source address that doesn't match the sender's IP address:

When pinging C from A, A sends an IP packet inside an Ethernet frame over tun0 to B which roughly looks like this:

----------------------------------------------------
| source: A's tun0 MAC | destination: B's tun0 MAC |
| -------------------------------------------------|
| | source: 10.8.0.4 | destination: 10.128.13.224 ||
| | payload: PING                                 ||
| -------------------------------------------------|
----------------------------------------------------

B then checks its routing and sends the following over eth1:

----------------------------------------------------
| source: B's eth1 MAC | destination: C's eth1 MAC |
| -------------------------------------------------|
| | source: 10.8.0.4 | destination: 10.128.13.224 ||
| | payload: PING                                 ||
| -------------------------------------------------|
----------------------------------------------------

The network inspects this and finds that the source address 10.8.0.4 does not match B's eth1 address 10.128.140.204 and doesn't transmit the frame. That's why a plain gateway won't work on this network.