1

First, a simple version of my network topology.

                      [ hub ]
                         |
                      OpenVPN
                         |
10.77.1.0/24--[ base ]---|---[ mr0  ]--10.77.2.0/24
10.77.3.0/24--[ mr1  ]---|---[ mr2  ]--10.77.5.0/24
10.77.5.0/24--[ mr3  ]---| ...

hub is an Ubuntu VPS running an OpenVPN server with a static public IP. base is a pfSense firewall at my house that has a constant location but variable IP. Each of the mr# nodes is a mobile router that does not have a consistent IP address or location, and may or may not be behind an additional NAT firewall. The listed CIDR networks are supernets directly attached to their router.

For the purpose of reading logs, here are my IPs, router IDs, and OSes:

host  ip           ospf id        os
hub   10.77.0.254  10.77.255.254  ubuntu
base  10.77.0.253  10.77.255.253  pfsense
mr0   10.77.0.252  10.77.255.252  lede
mr1   10.77.0.251  10.77.255.251  pfsense
mr2   10.77.0.250  10.77.255.250  centos
mr3   10.77.0.249  10.77.255.249  vyos

On the hub server, I have client-to-client enabled and lots of routes and iroutes pushed to the various clients. With this setup, I can connect from any node to any node (within the firewall rules). But, this setup not scalable and I have to restart the clients and server every time a route changes, so I'm working on replacing this static config with Quagga OSPF.

My problem appears similar to https://serverfault.com/questions/189739/tough-routing-problem-with-linux-quagga-and-openvpn, however, their accepted solution was to make everything static routed, which is what I'm trying to avoid.

So far, I have dynamic routing working between my Linux router (mr2), hub, and a random Linux desktop I added to the VPN to test 3 OSPF routers. My VyOS and LEDE routers have issues for reasons unrelated to this question. (edit: VyOS nightly works)


Now for the conundrum:

My two pfSense firewalls (one is base and the other is mr1) both spam the similar log messages:

2017/12/31 13:14:55 OSPF: Packet[DD]: Neighbor 10.77.255.250: Initial DBD from Slave, ignoring.
2017/12/31 13:15:00 OSPF: Packet[DD] [Slave]: Neighbor 10.77.255.254 packet duplicated.
2017/12/31 13:15:00 OSPF: *** sendmsg in ospf_write failed to 10.77.0.254, id 0, off 0, len 72, interface ovpnc1, mtu 1500: Network is unreachable
2017/12/31 13:15:00 OSPF: *** sendmsg in ospf_write failed to 10.77.0.254, id 0, off 0, len 108, interface ovpnc1, mtu 1500: Network is unreachable
2017/12/31 13:15:00 OSPF: Packet[DD]: Neighbor 10.77.255.250: Initial DBD from Slave, ignoring.
2017/12/31 13:15:00 OSPF: *** sendmsg in ospf_write failed to 10.77.0.250, id 0, off 0, len 52, interface ovpnc1, mtu 1500: Network is unreachable
2017/12/31 13:15:05 OSPF: Packet[DD] [Slave]: Neighbor 10.77.255.254 packet duplicated.

Since the pfSense boxes can't send out their OSPF replies, this leaves the network in this state:

hub# sh ip os ne

    Neighbor ID Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
10.77.255.250     1 Full/Backup       37.090s 10.77.0.250     tun0:10.77.0.254         0     0     0
10.77.255.251     0 ExStart/DROther   31.661s 10.77.0.251     tun0:10.77.0.254         0     0     0
10.77.255.253     0 ExStart/DROther   31.648s 10.77.0.253     tun0:10.77.0.254         0     0     0

Since it can't send anything, the pfSense node gets stuck:

base# sh ip os ne

Neighbor ID     Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
10.77.255.250     1 ExStart/Backup    36.505s 10.77.0.250     ovpnc1:10.77.0.253       0     0     0
10.77.255.251     0 2-Way/DROther     37.276s 10.77.0.251     ovpnc1:10.77.0.253       0     0     0
10.77.255.254     1 Exchange/DR       37.763s 10.77.0.254     ovpnc1:10.77.0.253       1     0     0

I'm honestly not sure if this is an OpenVPN configuration issue, a Quagga configuration issue, an OS configuration issue, or a bug in one or more components. Quagga seems to work all right when I just use ethernet links (as it happens, base, mr1, and mr3 are all sharing a physical location at the moment), so it appears that this is limited to using OSPF over OpenVPN.


EDIT 2018 JAN 06: Throughout this process, I noticed that the common denominator was that every working router was Linux-based. Scouring my netstat results for the millionth time, I discovered that my pfSense boxes implemented OpenVPN's tun adapter differently: in Linux, the tun adapter's address is set with a subnet mask, thus allowing nodes attached to the VPN directly to be contacted "without" a router hop. In FreeBSD, the tun adapter exposed as the internal point-to-point (which it is, even in Linux), and an IPv4 route is added that maps to the rest of the network. Can't remember if I did this or it happened automatically, but the goal I guess is now: a) how do I get FreeBSD/pfSense to create a TUN adapter that would behave like Linux allowing "direct" connections, or b) how do I get Quagga to allow its packets to be routed?

pfSense:

[root@base ~]# ifconfig ovpnc1
ovpnc1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    inet6 fe80::20c:29ff:feac:faf4%ovpnc1 prefixlen 64 scopeid 0x16
    inet 10.77.0.253 --> 10.77.0.225  netmask 0xffffffe0
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: tun openvpn
    Opened by PID 27659
[root@base ~]# netstat -nr4 | grep -e ovpnc1 -e link#22
10.77.0.224/27     10.77.0.225        UGS      ovpnc1
10.77.0.225        link#22            UH       ovpnc1
10.77.0.253        link#22            UHS         lo0
...
[root@base ~]# ping -r -c 2 10.77.0.254
PING 10.77.0.254 (10.77.0.254): 56 data bytes
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable

--- 10.77.0.254 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

Linux:

[root@mr2 ~]# ip a sh dev tun0
18: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none
    inet 10.77.0.250/27 brd 10.77.0.255 scope global tun0
       valid_lft forever preferred_lft forever
[root@nl3-mr2 ~]# ip ro | grep tun0
10.77.0.224/27 dev tun0 proto kernel scope link src 10.77.0.250
...
[root@mr2 ~]# ping -r -c 2 10.77.0.254
PING 10.77.0.254 (10.77.0.254) 56(84) bytes of data.
64 bytes from 10.77.0.254: icmp_seq=1 ttl=64 time=51.1 ms
64 bytes from 10.77.0.254: icmp_seq=2 ttl=64 time=51.2 ms

--- 10.77.0.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 51.117/51.199/51.281/0.082 ms

OpenVPN server.conf:

port 1194
management localhost 1194
proto udp
dev tun

ca ca.crt
cert server.crt
dh dh2048.pem
tls-auth ta.key 0
key-direction 0

auth SHA256
cipher AES-256-CBC

mode server
tls-server
topology subnet
push "topology subnet"
ifconfig 10.77.0.254 255.255.255.224
push "route-gateway 10.77.0.254"
client-to-client
config ccd/routes.conf
push "route 10.77.4.0 255.255.255.0 10.77.0.254 250"
client-config-dir ccd

user nobody
group nogroup

persist-key
persist-tun

status openvpn-status.log

verb 4

Example ccd file (this one is for base):

config ccd/routes_push.conf
ifconfig-push 10.77.0.253 255.255.255.224
iroute 10.77.1.0 255.255.255.0

ccd/routes.conf:

route 10.77.1.0 255.255.255.0 10.77.0.253 250
route 10.77.2.0 255.255.255.0 10.77.0.252 250
route 10.77.3.0 255.255.255.0 10.77.0.251 250
route 10.77.5.0 255.255.255.0 10.77.0.250 250
route 10.77.6.0 255.255.255.0 10.77.0.249 250

ccd/routes_push.conf:

push "route 10.77.1.0 255.255.255.0 10.77.0.253 250"
push "route 10.77.2.0 255.255.255.0 10.77.0.252 250"
push "route 10.77.3.0 255.255.255.0 10.77.0.251 250"
push "route 10.77.5.0 255.255.255.0 10.77.0.250 250"
push "route 10.77.6.0 255.255.255.0 10.77.0.249 250"

Abbreviated running config on hub:

...
!
service advanced-vty
service password-encryption
!
debug ospf event
!
...
!
interface tun0
 description grandcentral-hub
 ip ospf network broadcast
 ipv6 nd suppress-ra
 no link-detect
!
router ospf
 ospf router-id 10.77.255.254
 log-adjacency-changes
 network 10.77.0.224/27 area 0.0.0.0
 network 10.77.4.0/24 area 0.0.0.0
 network 172.17.0.0/16 area 0.0.0.0
 area 0.0.0.0 range 10.77.4.0/24
!
ip forwarding
ipv6 forwarding
!
line vty
!
end

Abbreviated running config on base, as generated by pfSense:

...
!
interface ovpnc1
 ip ospf network broadcast
 ip ospf priority 0
!
...
!
router ospf
 ospf router-id 10.77.255.253
 log-adjacency-changes detail
 passive-interface em0.1
 network 10.77.0.224/27 area 0.0.0.0
 network 10.77.1.0/27 area 0.0.0.0
 network 10.77.1.192/26 area 0.0.0.0
 network 192.168.254.0/24 area 0.0.0.0
!
ip forwarding
ipv6 forwarding
!
line vty
!
end

1 Answer 1

0

EDIT 20 January 2018 17:57 CST: as grawity pointed out (thank you very much), just using tap devices instead of tun fixes everything. Should have thought of that earlier.

That being said, I'm still looking for a solution with tun to see if it's possible. I don't know if ~20 odd bytes from the ethernet header might actually matter for a DSL link, and for sure the logging is more opaque since all the MULTI: Learn: entries are now virtual MAC addresses instead of IP addresses.


It appears that what I am trying to do isn't possible directly with tun. OpenVPN is a point-to-point interface by nature, and Linux's implementation of OpenVPN / TUN allows client-to-client communication to be hidden within the network stack.

By contrast, FreeBSD exposes the raw point-to-point interface and installs a route to handle client-to-client (tested on a clean FreeBSD 11.1-RELEASE).

Additionally, when you change the internal IP address of an OpenVPN server, it still assigns the default IP address (lowest available IP in the range) to point-to-point interfaces.

So, to fix this, I'll have to reimplement everything as a point-to-multipoint system (yay learning new things). I'll update this answer when I get it.

4
  • 1
    Perhaps you should use tap mode instead? Jan 20, 2018 at 19:16
  • I definitely should have thought of that earlier *facepalm*. I would prefer to use tun and not have to encapsulate the extra ethernet headers, but this works for now. Jan 20, 2018 at 23:54
  • 1
    You may need to configure the interface as an OSPF non-broadcast interface. Many tunnels do not support multicast, which is what OSPF will try to use on a broadcast network. That means that OSPF will need to be configured with a neighbor statement.
    – Ron Maupin
    Jan 21, 2018 at 0:35
  • That works as long as you don't change the VPN server's IP address and you don't need to form an adjacency with any router other than the VPN hub. Unfortunately, since it's not multicast that's broken in FreeBSD tun (Unicast is the weird one), you still get OSPF: *** sendmsg in ospf_write failed to 10.77.0.254, id 0, off 0, len 72, interface tun0, mtu 1500: Network is unreachable Jan 21, 2018 at 1:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .