I just need to re-route all user traffic directly to specified gateway (different from the default one) and bypass all VPN and other routing rules.
Let's start it from scratch (reboot the system),
No interface configured, no firewalled daemon works, just a clean system without any iptables rules.
reboot
[root@localhost ~]# cat /etc/iproute2/rt_tables|wc -l
0
[root@localhost ~]# ifconfig -a
enp3s0: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether xx:xx:xx:xx:xx:xx txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@localhost ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
[root@localhost ~]# ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
[root@localhost ~]# ip route
[root@localhost ~]# ip route show table all
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
unreachable default dev lo table unspec proto kernel metric 4294967295 error -101
local ::1 dev lo table local proto none metric 0
unreachable default dev lo table unspec proto kernel metric 4294967295 error -101
[root@localhost ~]#
[root@localhost ~]# ifconfig enp3s0 192.168.77.8/24 up
[root@localhost ~]# ip route
192.168.77.0/24 dev enp3s0 proto kernel scope link src 192.168.77.8
[root@localhost ~]# ip route show table all
192.168.77.0/24 dev enp3s0 proto kernel scope link src 192.168.77.8
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.77.0 dev enp3s0 table local proto kernel scope link src 192.168.77.8
local 192.168.77.8 dev enp3s0 table local proto kernel scope host src 192.168.77.8
broadcast 192.168.77.xxx dev enp3s0 table local proto kernel scope link src 192.168.77.8
fe80::/64 dev enp3s0 proto kernel metric 256
unreachable default dev lo table 0 proto kernel metric 4294967295 error -101
local ::1 dev lo table local proto none metric 0
local fe80::f279:59ff:fe6c:7833 dev lo table local proto none metric 0
ff00::/8 dev enp3s0 table local metric 256
unreachable default dev lo table 0 proto kernel metric 4294967295 error -101
Gateway respond to us:
[root@localhost ~]# ping -c 1 192.168.77.150
PING 192.168.77.150 (192.168.77.150) 56(84) bytes of data.
64 bytes from 192.168.77.150: icmp_seq=1 ttl=64 time=0.388 ms
--- 192.168.77.150 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.388/0.388/0.388/0.000 ms
[root@localhost ~]# iptables -L -nv
Chain INPUT (policy ACCEPT 16 packets, 2011 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 17 packets, 1966 bytes)
pkts bytes target prot opt in out source destination
[root@localhost ~]# iptables -L -nv -t nat
Chain PREROUTING (policy ACCEPT 1 packets, 125 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 1 packets, 125 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 3 packets, 241 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 3 packets, 241 bytes)
pkts bytes target prot opt in out source destination
[root@localhost ~]# telnet 8.8.8.8 53
Trying 8.8.8.8...
telnet: connect to address 8.8.8.8: Network is unreachable
[root@localhost ~]# iptables -I OUTPUT -m owner --uid-owner test1 -j MARK --set-mark 2
[test1@localhost ~]$ telnet 8.8.8.8 53
Trying 8.8.8.8...
telnet: connect to address 8.8.8.8: Network is unreachable
echo 2 novpn >> /etc/iproute2/rt_tables
[root@localhost ~]# echo 2 novpn >> /etc/iproute2/rt_tables
[root@localhost ~]# ip rule add fwmark 2 lookup novpn priority 2
[root@localhost ~]# ip rule
0: from all lookup local
2: from all fwmark 0x2 lookup novpn
32766: from all lookup main
32767: from all lookup default
[root@localhost ~]# ip route
192.168.77.0/24 dev enp3s0 proto kernel scope link src 192.168.77.8
[root@localhost ~]#
[root@localhost ~]# ip route add default via 192.168.77.150 dev enp3s0 table novpn
[test1@localhost ~]$ telnet 8.8.8.8 53
Trying 8.8.8.8...
telnet: connect to address 8.8.8.8: Network is unreachable
[test1@localhost ~]$
[root@localhost ~]# iptables -L -nv
Chain INPUT (policy ACCEPT 1 packets, 125 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1007 MARK set 0x2
[root@localhost ~]#
iptables rule stays unmatched, how to fix it?
Guessing, it looks like we need to define default gateway:
route add default gw 192.168.77.150
Testing it now:
[root@localhost ~]# iptables -L -nv
Chain INPUT (policy ACCEPT 3 packets, 375 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1007 MARK set 0x2
Counters not yet increased, it is OK.
[test1@localhost ~]$ telnet 8.8.8.8 53
Trying 8.8.8.8...
Connected to 8.8.8.8.
Escape character is '^]'.
Connection closed by foreign host.
Counters increased:
[root@localhost ~]# iptables -L -nv
Chain INPUT (policy ACCEPT 6 packets, 539 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 3 packets, 164 bytes)
pkts bytes target prot opt in out source destination
3 164 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1007 MARK set 0x2
[root@localhost ~]# ip route del default via 192.168.77.150 dev enp3s0 table novpn
[root@localhost ~]# show route table novpn
(no output)
[root@localhost ~]# ip route add default via 192.168.77.33 dev enp3s0 table novpn #192.168.77.33 does not exists
[test1@localhost ~]$ telnet 8.8.8.8 53
Trying 8.8.8.8...
Connected to 8.8.8.8.
Escape character is '^]'.
Connection closed by foreign host.
It should not to connect to 8.8.8.8, since 192.168.77.33 gateway does not exist.
So, the IP route rule doesn't work.
[root@localhost ~]# ip rule
0: from all lookup local
2: from all fwmark 0x2 lookup novpn
32766: from all lookup main
32767: from all lookup default
[root@localhost ~]# ip route show table novpn
default via 192.168.77.33 dev enp3s0
[root@localhost ~]# ip route show table main|fgrep default
default via 192.168.77.150 dev enp3s0
[root@localhost ~]# iptables -L OUTPUT -nv
Chain OUTPUT (policy ACCEPT 196 packets, 40014 bytes)
pkts bytes target prot opt in out source destination
9 492 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1007 MARK set 0x2
How to properly set up all per-user traffic via a different gateway?
Fedora release 22 (Twenty Two)
Linux lain 4.1.7-200.fc22.i686+PAE #1 SMP Mon Sep 14 20:36:56 UTC 2015 i686 i686 i386 GNU/Linux
iproute-3.16.0-3.fc22.i686
iptables-1.4.21-14.fc22.i686
