Routing works incorrectly with wireguard inside Docker network

Question

I'm not even sure that's the right wording, I'll try to explain.

I want to setup a small VPN are to host some private projects as well as access internet. I also want to have my own DNS server to have cool top level domain names.

How does my setup look

I am using two Docker images:

• ubuntu/bind9 for DNS
• donaldzou/wgdashboard (https://github.com/donaldzou/WGDashboard) for Wireguard and GUI dashboard

My wireguard interface has name main and runs on port 20000. Just a heads up because these aren't "default" settings.

Now, it almost works meaning I can connect several peers to this "server" peer. I can ping any peer from another peer. So the "wireguard" part of the setup works. I believe it is a routing issue that's outside wireguard scope.

Here's relevant parts of my docker-compose.yml file:

services:
  dns:
    image: ubuntu/bind9
    restart: unless-stopped
    networks:
      main:
        ipv4_address: 172.16.0.53 

  wgdashboard:
    image: donaldzou/wgdashboard
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    networks:
      main:
        ipv4_address: 172.16.0.2
    ports:
      - 10086:10086
      - 20000:20000/udp

networks:
  main:
    ipam:
      driver: default
      config:
        - subnet: 172.16.0.0/16
          ip_range: 172.16.0.0/24
          gateway: 172.16.0.1

Network is created and both containers have their correct IPs

So as my setup runs on UDP port 20000, and not on the host network, I have some rules in iptables. This part I'm super unsure about.

# iptables  -L | grep 20000
ACCEPT     udp  --  anywhere             172.16.0.2           udp dpt:20000
ACCEPT     udp  --  anywhere             anywhere             udp dpt:20000
# iptables -t nat -L | grep 20000
MASQUERADE udp  --  172.16.0.2           172.16.0.2           udp dpt:20000
DNAT       udp  --  anywhere             anywhere             udp dpt:20000 to:172.16.0.2:20000

I've also seen some people requested this network info, so I'm posting it here:

$ ip -br link
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> 
ens3             UP             52:54:00:0c:f3:c5 <BROADCAST,MULTICAST,UP,LOWER_UP> 
docker0          DOWN           02:42:61:07:a5:d4 <NO-CARRIER,BROADCAST,MULTICAST,UP> 
br-e12a65bf943c  UP             02:42:5b:d4:1a:a6 <BROADCAST,MULTICAST,UP,LOWER_UP> 
$ ip -br address
lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens3             UP             XX.YY.ZZ.24/24 
docker0          DOWN           172.17.0.1/16 
br-e12a65bf943c  UP             172.16.0.1/16 
$ ip route
default via XX.YY.ZZ.1 dev ens3 onlink 
XX.YY.ZZ.0/24 dev ens3 proto kernel scope link src XX.YY.ZZ.24 
172.16.0.0/16 dev br-e12a65bf943c proto kernel scope link src 172.16.0.1 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
$ ip rule
0:  from all lookup local 
32766:  from all lookup main 
32767:  from all lookup default 
$ 

What's wrong

• I can ping and query 172.16.0.53 (the "dns" container) from server that runs docker

• I can ping and query 172.16.0.53 from "wgdashboard" container

• I can ping and query 1.1.1.1 from server and both containers

• I can ping and open http://172.16.0.2:10086/ ("wgdashboard") container in browser from all connected peers

• I can NOT ping outside world from either peers

• I can NOT ping or issue a DNS query to 172.16.0.53 from either peers

---

What I'm trying to do to troubleshoot this problem:

port mapping

first, I added port mapping (53:53/udp) to dns service in docker-compose.yml:

$ docker-compose up
[+] Running 3/3
 ⠿ Network apps_main             Created                                                                                                                                                                                                                                                                0.1s
 ⠿ Container apps-dns-1          Created                                                                                                                                                                                                                                                                0.1s
 ⠿ Container apps-wgdashboard-1  Created                                                                                                                                                                                                                                                                0.1s
Attaching to apps-dns-1, apps-wgdashboard-1
Error response from daemon: driver failed programming external connectivity on endpoint apps-dns-1 (4d28f6ed5232a08b266aed9fd232bc9894b7ae0990d25129b27d4f4112ef289f): Error starting userland proxy: listen udp4 0.0.0.0:53: bind: address already in use

uh what, does it try to bind to port 53 of host machine? Anyway I shouldn't need this anyway, because I only plan to query this server from inside of wireguard (and therefore, docker) network and don't need it from outside?

tcpdump

I ran tcpdump from four locations:

• host machine, interface br-e12a65bf943c (docker-compose network), tcpdump -i br-e12a65bf943c icmp
• host machine, interface ens3 (actual internet), tcpdump -i ens3 icmp
• wireguard container, interface main (like wg0 but i gave it the different name), tcpdump -i main icmp
• bind9 container, interface eth0 (it's the only one there)

Pinging dns container from connected peer in chronological order (I don't know what's up with these timezones, ignore hours):

[wireguard,main]      16:18:22.021439 IP 10.0.0.2     > 172.16.0.53: ICMP echo request 
[host,br-e12a65bf943c]17:18:22.021500 IP 10.0.0.2     > 172.16.0.53: ICMP echo request 
[dns,eth0]            14:18:22.021508 IP 10.0.0.2     > 172.16.0.53: ICMP echo request 
[dns,eth0]            14:18:22.021567 IP 172.16.0.53  > 10.0.0.2:    ICMP echo reply   
[host,br-e12a65bf943c]17:18:22.021575 IP 172.16.0.53  > 10.0.0.2:    ICMP echo reply   
[host,ens3]           17:18:22.021584 IP 172.16.0.53  > 10.0.0.2:    ICMP echo reply   

takeaway: three requests,three replies, but one reply decided to go through ens3 instead of returning to wireguard

Pinging 1.1.1.1

[wireguard,main]      16:30:16.689460 IP 10.0.0.2 > 1.1.1.1: ICMP echo request
[host,br-e12a65bf943c]17:30:16.689570 IP 10.0.0.2 > 1.1.1.1: ICMP echo request

host machine, ens3 has nothing (!!!)

takeaway: I can't ping outside world from 10.0.0.0/16 network

So, I have three questions

• what happened with ICMP reply at br-e12a65bf943c? How to make it go back to wireguard interface?
• what happened with ICMP request on its way to 1.1.1.1?
• is it iptables

If you've read so far, thank you for your time

---

Solution, I guess?

Thanks for answering! You're rignt, I forgot to mention the network mask, I selected /16 in this case.

Indeed, adding this route on dns container fixed pinging

ip route add 10.0.0.0/16 via 172.16.0.2

Now my task is to make it "permanent", I guess I need to add this to dockerfile or something.

Second issue is external traffic.

I added this rule to wireguard container:

iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -o eth0 -j MASQUERADE

And yeah it looks like it's working

Answer 1

First of all, you should have a route for 10.0.0.0/24 (well, /24 assumed since unprovided) on the DNS container, with 172.16.0.2 as the nexthop (a.k.a. gateway). Alternatively, you'll need to (S)NAT (a.k.a. IP masquerade) inside the WG container (for at least traffics destined for 172.16.0.53:53, UDP and maybe also TCP).

As for traffics to the Internet (e.g. 1.1.1.1), it's probably because the firewall rules on the docker host added by the docker daemon do not allow traffics with source address other than 172.16.0.0/16 (subnet:) to be forwarded out from the br-e12a65bf943c. Again one of the options is to (S)NAT inside the WG container for traffics with source address in 10.0.0.0/24.

The other option is to disable the corresponding filtering by some means, like preventing the docker daemon from touching the firewall at all (--iptables=false), or having something add the exception after the docker daemon is started.

I'm not entirely sure whether you'll need to add (S)NAT rule specifically for 10.0.0.0/24 on the docker host if you decide not to have the additional layer of NAT inside the WG container. You can check with iptables-save to see whether the current SNAT/MASQUERADE rule on the docker host is e.g. limited to source address within 172.16.0.0/16 for traffics from br-e12a65bf943c. Also with that decision you'll need to add route for 10.0.0.0/24 (with 172.16.0.2 as the nexthop) on the docker host as well.