0

Let's say we have machines with two interfaces: ens160 and wg0.

Let's say we have routes like below: 172.29.248.128/25 dev ens160 proto kernel scope link src 172.29.248.134 192.168.2.0/24 dev wg0 proto kernel scope link src 192.168.2.2 Our first machine has ips: 172.29.248.134, 192.168.2.2

There's second machine: endpoint: 172.29.244.20:51820 allowed ips: 192.168.2.1/32

First machine can connect to second machine on both IPs, 172.29.244.20 and 192.168.2.1.

Now, what I want to achieve is, when first machine is trying to reach 172.29.244.20, it should go through wg0 interface. Something like destination IP rewrite for this specific IP, from 172.29.244.20 to 192.168.2.1

Is it possible? If yes, can we also rewrite it on second machine, so if it receives anything on wg0 interface from 192.168.2.2, source ip should be changed back to 172.29.248.134 and passed to ens160?

Can it be done purely with ip route? Or do we need iptables?

@dirkt Thanks, that would normally work and I tried, but it seems situation is little more complicated. Actually whole more complicated, because i'm trying to setup encryption between machines for Kubernetes cluster, but due to problems with network plugin (Calico), in some more transparent way than directly using 192.168.2.0/24. Basically there's IP-in-IP encapsulation over layer 3 VPN tunnel and a lot of iptables rules. wg0 is Wireguard (VPN) interface, it has 192.168.2.0/24 subnet. I use 192.168.2.1/32 to specify single IP for VPN, because I want to create mesh network. I can't specify 192.168.2.0/24 for allowed IPs, because wireguard does not allow same subnet for multiple peers.

Now i see that forwarding all trafic to this ip via wg0 might be dumb idea, because I can see with tcpdump on machine sending ping works correctly (ICMP requests are visible), but nothing shows on receiving machine wg0.

I asked about it on wireguard IRC and the problem probably is: The problem is that outgoing traffic to the other wireguard endpoint is no longer routed over ens160. But looped into the wireguard interface.

Okay, but there might be a solution here, from wireguard documentation . FwMark — a 32-bit fwmark for outgoing packets. If set to 0 or "off", this option is disabled. May be specified in hexadecimal by prepending "0x". Optional.

I found article about this "recursive VPN problem", but my networking skills are quite poor. What should I do to route traffic to 172.29.244.20/32 via 192.168.2.2 dev wg0 except packets marked with specific fwmark?

  • I still don't understand the complete situation, but the problem seems related to Wireguard. Any reason you have to use Wireguard "with a lot of iptables rules" (which you told us nothing about, and which are probably very important to solve the problem), and can't use a different tunnel? E.g. OpenVPN allows routing of segments (without iptables rules), though you have to set it up specifically. – dirkt Mar 25 '18 at 4:46
  • Can you complete some informations? routers (between LANs 172.29.244.0/??? <- which netmask? and 172.29.248.128/25) etc. so with the output of ip -br link; ip -br addr; ip route; wg (without -br if it fails, and feel free to edit) on both hosts, that should be fine. That's some glue information needed to work on a solution. Also do you intend to route just the two involved peers or also other hosts through this tunnel (might get more difficult) ? – A.B Mar 28 '18 at 20:04
  • pastebin.com/WZbukB86 , 172.29.244.0/25 and 172.29.248.128/25 . Optimally I would want only direct communication between each peer, so they don't need to forward to any other host. – Adiqq Mar 29 '18 at 8:12
  • I also tried marking these wireguard packets with fwmark and added ip rules to route marked packages through ens160, but it didn't help either. – Adiqq Mar 29 '18 at 8:16
  • You should probably include parts of the pastebin in the question. It started from two peers across two LANs, to a lot of peers in the same LAN in addition to across LANs. The initial solution was working fine for two peers. I had to rework it a bit to check some corner cases. There's no guarantee that there won't be interaction, either with Docker (which uses iptables and in some settings can set marks) or bird if it uses custom route policies itself. – A.B Mar 30 '18 at 3:59
1

Problem and goal

Having packets part of the WireGuard envelope (the UDP packets transporting the encrypted payload) not being subject to tunnel encapsulation themselves despite using the same IP inside the tunnel as outside the tunnel.

Method

Because the routing stack (ip route, ip rule...) is working at layer 3 (IP), and not at layer 4 (UDP, used by envelope), this can't be solved only with the routing stack without extra help to distinguish envelope packets from "data" packets.

Luckily it's possible to tag a packet with a mark (purely internally, on the skbuff of the packet, of course not on the wire). The tag can then be reused as fwmark for the routing stack. The facilities (I know of) able to mark packets are iptables, nftables ... and WireGuard itself.

Since Wireguard nevertheless can mark packets after encapsulation there's no need to use iptables/nftables as extra help for the routing.

There's currently an annoying limitation in wireguard (which certainly exists for good reasons probably related to its routing handling): I couldn't find a way to set two peers with the same allowed-ips values. So it's not possible to simply set:

AllowedIps = 172.29.244.0/25,172.29.248.128/25

on every endpoint. Trying this will delete the allowed-ips on the previous endpoints with a same value. Setting any other overlapping range (eg changing the mask), will be accepted, but will prevent some of the overlapping peers to work. Of course allowing only the single remote endpoint's IP does work. That's the only way for the whole mesh to work correctly.

Solution

Distinguish between envelope packets and tunneled packets with a mark. There's not even need to state which UDP port it has to be, WireGuard handles this part. Use this mark to do an "exception to the exception" for the route choice.

Choices made

As per the OP's comment, there's no packet forwarding needed, so no forwarding was tested.

The choice made here is to mark packets that will not enter the tunnel with the help of WireGuard (an other working solution, using iptables instead of WireGuard to mark packets, could do the opposite and mark the packets entering the tunnel).

Instructions

Configuration example:

# ip -4 -br addr show dev ens160
ens160@if7       UP             172.29.248.132/25 
# ip route
default via 10.0.3.1 dev eth0 
10.0.3.0/24 dev eth0 proto kernel scope link src 10.0.3.147 
172.29.244.0/25 via 172.29.248.129 dev ens160 
172.29.248.128/25 dev ens160 proto kernel scope link src 172.29.248.132 

# wg show wg0
interface: wg0
  public key: 7FsVTBuIDNQsWj9flHWwMt+kCaE8urEE9bCwcvz6CBM=
  private key: (hidden)
  listening port: 51820

peer: +NJdQr+piwid1iuC58rbnXFgGo/vZ9d8Gs3xvl3/TxI=
  endpoint: 172.29.248.134:51820
  allowed ips: (none)
  transfer: 14.74 KiB received, 34.35 KiB sent

peer: KrZWEupjHGoHTywwWhg4XXQSyFl/disav7/pyhOK/1Q=
  endpoint: 172.29.244.20:51820
  allowed ips: (none)
  transfer: 11.06 KiB received, 15.65 KiB sent

Since in the end everything can go through the tunnel reusing the same IPs, the network 192.168.2.0/24 is not used anymore. Since no routing rule will reference those IPs, they can even be removed: wg0 will have zero IP on it (it's still ok to keep them if wished):

# ip addr flush dev wg0

Configure wg0 to mark envelope packets with some arbitrarily chosen value (here 9):

# wg set wg0 fwmark 9

Configure every WireGuard host to accept the endpoint's IP in the tunnel, that's the goal:

# wg show wg0|awk 'BEGIN { RS="\n\n" } /^peer/ { printf "wg set wg0 peer %s allowed-ips %s\n",$2,$4 }'|cut -d: -f1 | sh

Which now gives:

# wg show wg0
interface: wg0
  public key: 7FsVTBuIDNQsWj9flHWwMt+kCaE8urEE9bCwcvz6CBM=
  private key: (hidden)
  listening port: 51820
  fwmark: 0x9

peer: +NJdQr+piwid1iuC58rbnXFgGo/vZ9d8Gs3xvl3/TxI=
  endpoint: 172.29.248.134:51820
  allowed ips: 172.29.248.134/32
  transfer: 14.74 KiB received, 34.35 KiB sent

peer: KrZWEupjHGoHTywwWhg4XXQSyFl/disav7/pyhOK/1Q=
  endpoint: 172.29.244.20:51820
  allowed ips: 172.29.244.20/32
  transfer: 11.06 KiB received, 15.65 KiB sent

Add routes on (arbitrarily chosen) routing table 200, specifying host's IP as source (to avoid an IP from an unrelated interface being unfortunately chosen)...

...either only once, routes for both LANs, but this will require using WireGuard for any IP traffic with any peer in the LANs:

# localip=$(ip -4 -br addr show dev ens160|awk '{ print $3 }'|cut -d/ -f1)
# ip route add 172.29.244.0/25   dev wg0 table 200 src $localip
# ip route add 172.29.248.128/25 dev wg0 table 200 src $localip

Which gives:

# ip route show table 200
172.29.244.0/25 dev wg0 scope link src 172.29.248.132 
172.29.248.128/25 dev wg0 scope link src 172.29.248.132 

...either add per peer routes:

# localip=$(ip -4 -br addr show dev ens160|awk '{ print $3 }'|cut -d/ -f1)
# wg show wg0|awk 'BEGIN { RS="\n\n" } /^peer/ { printf "ip route add %s dev wg0 table 200 src %s\n",gensub(":.*$","",1,$4),localip }' localip=$localip | sh

Which gives instead here:

# ip route show table 200
172.29.244.20 dev wg0 scope link src 172.29.248.132 
172.29.248.134 dev wg0 scope link src 172.29.248.132 

Correct in advance the chicken-and-egg problems that would be introduced by the rule after this one, by using directly the main routing table for packets marked by WireGuard (those 51820/UDP packets):

# ip rule add priority 32000 fwmark 9 lookup main

"Insert" after a lookup to table 200. Actually only the packets with the routes set above for WireGuard will match and will use the tunnel through wg0, other packets will continue to the (usual) main table:

# ip rule add priority 32100 lookup 200

That's these rule 32100 and table 200 that replace the usual routing done inside wg0 when IPs are set on it.

Which now gives:

# ip rule
0:  from all lookup local 
32000:  from all fwmark 0x9 lookup main 
32100:  from all lookup 200 
32766:  from all lookup main 
32767:  from all lookup default 

Alas the reverse path filter is now triggered for the incoming non-WireGuard traffic coming from the normal interface (here ens160) when wg0 would now be expected. This includes ARP reply packets needed to find peers in the same LAN as well as the gateway to the other LAN (I don't have a good explanation on this). So rp_filter has to be set to loose mode for the normal interface (here ens160):

echo 2 > /proc/sys/net/ipv4/conf/ens160/rp_filter

With this settings repeated on the other peers, all peers can now communicate between each other using their own IP like if there was no tunnel (in a mesh potentially using up to n*(n-1)/2 tunnels).

Example: the host receives a single ping and replies:

# tcpdump -n -s0 -i wg0 -p
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
05:22:51.553463 IP 172.29.244.20 > 172.29.248.132: ICMP echo request, id 2083, seq 1, length 64
05:22:51.553525 IP 172.29.248.132 > 172.29.244.20: ICMP echo reply, id 2083, seq 1, length 64

While on the router the encrypted traffic seen is (with some overhead after probably because there was no recent activity):

# tcpdump -n -s0 -i br248 -p
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br248, link-type EN10MB (Ethernet), capture size 262144 bytes
05:22:51.553317 ARP, Request who-has 172.29.248.132 tell 172.29.248.129, length 28
05:22:51.553358 ARP, Reply 172.29.248.132 is-at 2e:02:6e:cf:6d:4f, length 28
05:22:51.553383 IP 172.29.244.20.51820 > 172.29.248.132.51820: UDP, length 128
05:22:51.553583 IP 172.29.248.132.51820 > 172.29.244.20.51820: UDP, length 128
05:22:51.554112 IP 172.29.248.132.51820 > 172.29.244.20.51820: UDP, length 148
05:22:51.555198 IP 172.29.244.20.51820 > 172.29.248.132.51820: UDP, length 92
05:22:51.555748 IP 172.29.248.132.51820 > 172.29.244.20.51820: UDP, length 32
0

You left out a complete description of your network, so I'll assume host A (the first one) and host B (the second one) are directly connected to the same segment on wg0 on host A with 192.168.2.2/24, and host B with 92.168.2.1/?? (the /32 you have is likely a mistake - each host on the segment should have the same netmask, so likely you mean /24). If this is not correct, please edit your question with a description of your network, netmasks, and IPs at endpoints.

Next, if you want to route, use the routing features of the kernel. I don't know why so many people think that iptables is for routing. It's not, no matter what you read on the internet. If normal routing is not enough, use policy routing. If policy routing is not enough, then you can think about doing funny stuff with iptables.

The routing features of the kernel are optimized to make routing decisions quickly. That's not true for iptables.

(Sorry for the rant, but that's a pet peeve of mine).

So in your case, all you have to do is to add a route on A to tell A that it should route via 192.168.2.2 when trying to contact 72.29.244.20:

ip route add 72.29.244.20/32 via 192.168.2.2 dev wg0

Then the packets with destination address 72.29.244.20 will be sent to 192.168.2.2, i.e. host B. When host B receives those, it will find out that it already has an interface with this address, and accept them.

You may even think of widening 72.29.244.20/32 to 72.29.244.0/24 or whatever, if there are other hosts in that range that can more conveniently be reached along this route.

  • OP forgot to tell that wg0 is a wireguard tunnel interface with probably a few specific settings ( wireguard.com/quickstart allowed-ips matters there) + probable tunnel end point chicken-and-egg consideration – A.B Mar 24 '18 at 21:36
  • I added some further explanation – Adiqq Mar 24 '18 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.