OpenVPN policy-based routing not working correctly

Question

I'm trying to setup an OpenVPN client on ArchLinux with PrivateInternetAccess. By default PIA pushes routes such that all traffic goes through the VPN. I only want some applications to use the VPN.

To do this, I'm using "policy based routing". I've created a new routing table named "vpn", and will selectively send users to that routing table.

After these commands, the "media" user will be routed to the "vpn" routing table:

$ echo 100 vpn >> /etc/iproute2/rt_tables
$ iptables -t mangle -I OUTPUT -m owner --uid-owner media -j MARK --set-mark 0x1
$ ip rule add fwmark 0x1 table vpn

I've additionally modified my OpenVPN client configuration so that my routing tables will get populated correctly:

$ cat /etc/openvpn/client.conf
    client
    dev tun
    proto udp
    remote us-newyorkcity.privateinternetaccess.com 1194
    resolv-retry infinite
    nobind
    persist-key
    persist-tun
    ca /etc/openvpn/ca.crt
    tls-client
    remote-cert-tls server
    auth-user-pass /etc/openvpn/login.conf
    comp-lzo
    verb 1
    reneg-sec 0
    crl-verify /etc/openvpn/crl.pem

    # This will override PIA so that traffic will route through our normal gateway
    route 0.0.0.0 192.0.0.0 net_gateway
    route 64.0.0.0 192.0.0.0 net_gateway
    route 128.0.0.0 192.0.0.0 net_gateway
    route 192.0.0.0 192.0.0.0 net_gateway

    # Calling these scripts will add the PIA routes to the vpn table
    script-security 2
    up /etc/openvpn/up.sh
    down /etc/openvpn/down.sh

$ cat /etc/openvpn/up.sh
    #!/bin/sh
    ip route add table vpn default via $ifconfig_remote

$ cat /etc/openvpn/down.sh
    #!/bin/sh
    ip route flush table vpn

Here are my routing tables after running openvpn /etc/openvpn/client.conf:

$ ip route show table main
    0.0.0.0/2 via 192.168.1.1 dev eth0
    0.0.0.0/1 via 10.197.1.5 dev tun0
    default via 192.168.1.1 dev eth0  src 192.168.1.124  metric 202
    10.197.1.1 via 10.197.1.5 dev tun0
    10.197.1.5 dev tun0  proto kernel  scope link  src 10.197.1.6
    64.0.0.0/2 via 192.168.1.1 dev eth0
    128.0.0.0/2 via 192.168.1.1 dev eth0
    128.0.0.0/1 via 10.197.1.5 dev tun0
    192.0.0.0/2 via 192.168.1.1 dev eth0
    192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.124  metric 202
    209.95.50.86 via 192.168.1.1 dev eth0

$ ip route show table vpn
    default via 10.197.1.5 dev tun0

Things run normally as a user that DOES NOT get routed to the "vpn" table:

$ whoami
    jordan

$ ping -c 2 8.8.8.8
    PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=38.9 ms
    64 bytes from 8.8.8.8: icmp_seq=2 ttl=46 time=39.0 ms

    --- 8.8.8.8 ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1001ms
    rtt min/avg/max/mdev = 38.952/38.999/39.047/0.203 ms

However, things are failing for a user that DOES get routed to the "vpn" table:

$ whoami
media

$ ping -c 2 8.8.8.8
    PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

    --- 8.8.8.8 ping statistics ---
    2 packets transmitted, 0 received, 100% packet loss, time 999ms

I took a look at tcpdump -i tun0 to see what's going on. It seems that icmp request are going out via tun0, but not coming back?

$ tcpdump -i tun0
    listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
    15:37:30.134399 IP keep > google-public-dns-a.google.com: ICMP echo request, id 10256, seq 1, length 64
    15:37:31.143217 IP keep > google-public-dns-a.google.com: ICMP echo request, id 10256, seq 2, length 64

Thoughts? :(

==== EDIT #1 ====

As a sanity check, if we send all traffic through the VPN (by removing the "route x.x.x.x 192.0.0.0 net_gateway" lines in /etc/openvpn/client.conf), we get icmp responses just fine:

$ tcpdump -i tun0
    listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
    16:26:54.401732 IP keep > google-public-dns-a.google.com: ICMP echo request, id 10480, seq 1, length 64
    16:26:54.483122 IP google-public-dns-a.google.com > keep: ICMP echo reply, id 10480, seq 1, length 64
    16:26:55.403465 IP keep > google-public-dns-a.google.com: ICMP echo request, id 10480, seq 2, length 64
    16:26:55.485068 IP google-public-dns-a.google.com > keep: ICMP echo reply, id 10480, seq 2, length 64

==== EDIT #2 ====

Following MariusMatutiae 's advice, I tried using the --route-noexec flag to manually set the routes myself in /etc/openvpn/up.sh. We also disable reverse path filtering in /etc/openvpn/up.sh and re-enable it in /etc/openvpn/down.sh:

$ cat /etc/openvpn/up.sh
    #!/bin/sh
    ip route add table vpn 0.0.0.0/1 via $ifconfig_remote
    ip route add table vpn 128.0.0.0/1 via $ifconfig_remote
    ip route add table vpn $route_network_1 via $ifconfig_remote
    ip route add table vpn $trusted_ip via $route_net_gateway

    ip route add table vpn $ifconfig_remote dev tun0 proto kernel src $ifconfig_local
    ip route add table vpn 192.168.1.0/24 dev eth0 proto kernel src 192.168.1.124 metric 202

    ip route del table main $ifconfig_remote

    # Disable reverse path filtering
    for f in /proc/sys/net/ipv4/conf/*/rp_filter; do
            echo 0 > $f;
    done

$ cat /etc/openvpn/down.sh
    #!/bin/sh
    ip route flush table vpn

    # Re-enable reverse path filtering
    for f in /proc/sys/net/ipv4/conf/*/rp_filter; do
            echo 1 > $f;
    done

Afterwards, my routing tables look like this:

$ ip route show table main
    default via 192.168.1.1 dev eth0  src 192.168.1.124  metric 202
    192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.124  metric 202

$ ip route show table vpn
    0.0.0.0/1 via 10.173.1.5 dev tun0
    10.173.1.1 via 10.173.1.5 dev tun0
    10.173.1.5 dev tun0  proto kernel  scope link  src 10.173.1.6
    128.0.0.0/1 via 10.173.1.5 dev tun0
    192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.124  metric 202
    209.95.50.133 via 192.168.1.1 dev eth0

However, user "media" is still unable to ping 8.8.8.8. tcpdump -i tun0 still reports that no responses are coming back :(

Answer 1

You have incorrectly partitioned your routes thru your two routing tables. The following routes, appearing in the main table, belong to the vpn table:

0.0.0.0/1 via 10.197.1.5 dev tun0
128.0.0.0/1 via 10.197.1.5 dev tun0
10.197.1.1 via 10.197.1.5 dev tun0     
10.197.1.5 dev tun0  proto kernel  scope link  src 10.197.1.6
209.95.50.86 via 192.168.1.1 dev eth0

The following route belongs to both tables:

192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.124  metric 202

The reason for this is that: the first four rules involve the interface tun0, the virtual NIC which is brought up by OpenVPN, which clearly has nothing to do with the main table. The fifth rule teaches your kernel how to reach your PIA server. The last rule allows both tables to have access to local pcs, things like printers, NAS, and so on.

As for how to set this up automatically, this is made slightly complex by the fact that your OpenVPN server may assign you a tun0 IP address which differs every time: since you do not control the server, you may not configure it to assign your tun0 interface a static address.

So what you need to do is to learn to use, first, the option --route-no-exec. The Manual states:

--route-no-exec

Don't add or remove routes automatically. Instead pass routes to --route-up script using environmental variables.

Then you should learn how to use the Environmental Variables (there is a whole section at the end of the Manual linked above), especially those called ifconfig--something.

Also, you will find it necessary to disable reverse path filtering by the kernel,

for f in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 0 > $f; done

I would place this command in your up script, and the reverse one

for f in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 1 > $f; done

in your down script. This is necessary because you have a default gateway on the main table, which means that, when the kernel checks whether there is some spoofing, any packet is flagged.

Alternatively, and more easily, you may consider switching the role of the two tables: let OpenVPN do its thing automatically, and setup the routes for the main routing table, and setup a new routing table called non-vpn which has your usual configuration,

# ip route show
  default via 192.168.0.1 dev eth0  proto static 
  192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.74  metric 1 

(or something like this) for the packets not matching the MARK above. This is much easier to setup.

Answer 2

I gave up with getting policy-based routing to work. Instead, I ended up using network namespaces to accomplish my goal. A good guide to get started can be found here https://schnouki.net/posts/2014/12/12/openvpn-for-a-single-application-on-linux/

Special thanks to MariusMatutiae for all the help!