0

! See updated post at the end !

I am trying to run Wireguard in a container using docker compose on a remote host in a site-to-site configuration with my intranet at home which works flawlessly on its own.

Wireguard configuration

wg0.conf

[Interface]
PrivateKey = ***
Address = 10.6.0.4/24
ListenPort = 12345
DNS = 208.67.222.222, 208.67.220.220
PostUp = iptables -t nat -A POSTROUTING -o wg+ -j MASQUERADE
PreDown = iptables -t nat -D POSTROUTING -o wg+ -j MASQUERADE

[Peer]
PublicKey = LXSxUv5lp9A2WOz5mV33GQa5jpJYJ04j4Rl6FWlnczA=
PresharedKey = ***
Endpoint = vpn.example.com:12345
AllowedIPs = 192.168.178.0/24, 10.6.0.0/24

Docker compose file

docker-compose.yml

version: '3.*'

networks:
  outside:
    external: true

  wireguard_ghf68:
    internal: true
    driver: "bridge"
    ipam:
      config:
        - subnet: 10.7.3.0/24

services:
  wireguard:
    image: lscr.io/linuxserver/wireguard:latest
    container_name: wireguard

    cap_add:
      - NET_ADMIN
      - SYS_MODULE

    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Berlin

    volumes:
      - './wireguard:/config'
      - '/lib/modules:/lib/modules:ro'

    networks:
      wireguard_ghf68:
       ipv4_address: 10.7.3.3
      outside: {}

    ports:
      - target: 51820
        published: 51820
        protocol: "udp"
        mode: "host"

    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv4.ip_forward=1

    restart: unless-stopped

After a docker compose up I can attach a shell to the container and ping all the endpoints in my intranet (192.168.178.0/24).

root@wireguard:/# ping -c1 192.168.178.25
PING 192.168.178.25 (192.168.178.25) 56(84) bytes of data.
64 bytes from 192.168.178.25: icmp_seq=1 ttl=63 time=47.6 ms

--- 192.168.178.25 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 47.640/47.640/47.640/0.000 ms

Docker compose file with homeassistant

docker-compose.yml:

services:
  homeassistant:
    container_name: "homeassistant"
    image: "ghcr.io/home-assistant/home-assistant:stable"

    depends_on:
     - "wireguard"

    cap_add:
      - NET_ADMIN

    volumes:
      - ./homeassistant:/config
      - /etc/localtime:/etc/localtime:ro

    environment:
      - PUID=1000
      - PGID=1000
      - "TZ=Europe/Berlin"

    restart: unless-stopped
    networks:
      wireguard_ghf68:
        ipv4_address: 10.7.3.2

After docker compose up I can attach a shell to "homeassistant" and ping the wireguard container. But I also want to be able to ping my intranet (192.168.178.0/24) from there. So I create a new route inside the container:

ip -4 route add 192.168.178.0/24 via 10.7.3.3

which results in these routes:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.7.3.1        0.0.0.0         UG    0      0        0 eth0
10.7.3.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.178.0   10.7.3.3        255.255.255.0   UG    0      0        0 eth0

Now I theoretically should be able to ping my intranet but it does not work.

root@homeassistant:/# ping -c1 10.7.3.3
PING 10.7.3.3 (10.7.3.3): 56 data bytes
64 bytes from 10.7.3.3: seq=0 ttl=64 time=0.147 ms

--- 10.7.3.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.147/0.147/0.147 ms

root@homeassistant:/# ping -c1 192.168.178.1
PING 192.168.178.1 (192.168.178.1): 56 data bytes

--- 192.168.178.1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

Docker Host

However if I create the same route on the docker host I am able to reach my intranet from that host but still not from within the "homeassistant" container.

root@dockerhost:~$ ip -4 route add 192.168.178.0/24 via 10.7.3.3
root@dockerhost:~$ ping -c1 192.168.178.25
PING 192.168.178.25 (192.168.178.25) 56(84) bytes of data.
64 bytes from 192.168.178.25: icmp_seq=1 ttl=62 time=43.8 ms

--- 192.168.178.25 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 43.826/43.826/43.826/0.000 ms

Questions

What am I doing wrong? Am I missing some routes or sysctl configurations or something in cap_add? Why is it working from the docker host but not from the within the container?

If you need more information please ask.

Update

I finally found the culprit. When docker creates a network it also creates these iptables rules:

-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
...
-A DOCKER-ISOLATION-STAGE-1 ! -s 10.7.3.0/24 -o br-a575507e42d8 -j DROP
-A DOCKER-ISOLATION-STAGE-1 ! -d 10.7.3.0/24 -i br-a575507e42d8 -j DROP
...
-A DOCKER-USER -j RETURN

After removing the two DROP rules the routing works as expected.

I think the better way would be to add a few rules to the DOCKER-USER chain which accept the packets before they can be dropped.

If I find a elegant way to add these rules dynamically I will answer my question myself.

1 Answer 1

0

I will now answer my own question because I found a way which seems to be the best for me at the moment.

First I give the docker network a static bridge name, in this case d-wg (short for docker-wireguard). Keep in mind that there is a maximum length of 15 characters or 16 bytes including the null character (see if.h)

networks:
  wireguard:
    driver: "bridge"
    internal: true
    name: "wireguard"
    ipam:
      driver: "default"
      config:
        - subnet: 10.7.3.0/24
    driver_opts:
      com.docker.network.bridge.name: d-wg

Now a docker compose up will create a new docker network with the name wireguard and the interface name on kernel level will be d-wg:

$ docker network ls -f name=wireguard
NETWORK ID     NAME        DRIVER    SCOPE
648c3d39638b   wireguard   bridge    local

$ ip link show d-wg
289: d-wg: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:43:xx:yy:zz brd ff:ff:ff:ff:ff:ff

Now I can easily create two iptables rules in the DOCKER-USER chain which allow traffic for my two subnets 10.7.3.0/24 on the local side and 192.168.178.0/24 on the remote side. Then make sure the already existing rule -A DOCKER-USER -j RETURN goes after these again:

$ iptables -A DOCKER-USER -d 192.168.178.0/24 -i d-wg -o d-wg -j ACCEPT
$ iptables -A DOCKER-USER -d 10.7.3.0/24 -i d-wg -o d-wg -j ACCEPT
# Add a new return rule
$ iptables -A DOCKER-USER -j RETURN
# Delete the old one (deletes always the first if there are multiple rules)
$ iptables -D DOCKER-USER -j RETURN

Now I also have to create a new route inside my homeassistant container when it is starting up. Its original entrypoint is /init and I want to create the new route to the wireguard container before that. Therefore I create a small script init-script.sh which will be the new entrypoint and mount it into the container: init-script.sh

#!/bin/bash

ip route add 192.168.178.0/24 via 10.7.3.3
# 'exec' is necessary to not spawn a new process with an other PID than 1.
exec /init

And this is my new docker-compose.yml for homeassistant.

services:
  homeassistant:
    container_name: "homeassistant"
    image: "ghcr.io/home-assistant/home-assistant:stable"

    cap_add:
      - NET_ADMIN

    volumes:
      - ./homeassistant:/config
      - ./init-script.sh:/init-script.sh:ro
      - /etc/localtime:/etc/localtime:ro

    environment:
      - PUID=1000
      - PGID=1000
      - "TZ=Europe/Berlin"

    entrypoint:
      - /init-script.sh

    restart: unless-stopped

    networks:
      traefik:
      wireguard:
        ipv4_address: 10.7.3.2

An additional idea would be to create a new systemd service which creates the iptables rules automatically after the docker service starts. Maybe I will edit this answer as soon as I did this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.