I have sooo struggled with this so here's a COMPLETE solution. It's tested on Ubuntu 15 to 19.10. You can especially use it with OpenVPN to route certain apps outside the VPN tunnel interface.
The complete "cgroup" solution
How that works?
- Linux kernel will put the app into a control group. Network traffic from apps in this cgroup will be identified by their class ID at the network controller level.
- iptables will mark this traffic and force it to exit with the right IP
- ip route will process marked traffic in a different routing table, with a default route to whatever gateway IP you want.
I've made a novpn.sh script to automate dependencies install & run. Tested on Ubuntu 15 to 19.10.
Start your VPN first.
# If you don't use eth0, edit the script setting.
sudo chmod +x novpn.sh
./novpn.sh traceroute www.google.com
First, install cgroup support and tools:
sudo apt-get install cgroup-lite cgroup-tools
You need iptables 1.6.0+. Get iptables 1.6.0 release source, extract it, then run this (
--disable-nftables flag will avoid errors) from iptables source dir:
sudo apt-get install dh-autoreconf bison flex
./configure --prefix=/usr \
sudo make install
Now, the real config. Define a control group named
novpn. Processes in this cgroup will have a classid of
echo 0x00110011 > net_cls.classid
Now, we'll suppose the real interface you want to use for the specific app is
eth0 with a gateway IP of
10.0.0.1. REPLACE these by what you really want (get the info from
ip route), especially in newer Ubuntu versions where interfaces have weird names. Run still as root:
# Add mark 11 on packets of classid 0x00110011
iptables -t mangle -A OUTPUT -m cgroup --cgroup 0x00110011 -j MARK --set-mark 11
# Force the packets to exit through eth0 with NAT
iptables -t nat -A POSTROUTING -m cgroup --cgroup 0x00110011 -o eth0 -j MASQUERADE
# Define a new "novpn" routing table
# DO THIS JUST ONCE !
echo 11 novpn >> /etc/iproute2/rt_tables
# Packets with mark 11 will use novpn
ip rule add fwmark 11 table novpn
# Novpn has a default gateway to the interface you want to use
ip route add default via 10.0.0.1 table novpn
# Unset reverse path filtering for all interfaces, or at least for "eth0" and "all"
for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 0 > $i; done
Finally, run your app on the specific interface:
sudo cgcreate -t $USER:$USER -a $USER:$USER -g net_cls:novpn
cgexec -g net_cls:novpn traceroute www.google.com
# Close all Firefox windows first
killall firefox; cgexec -g net_cls:novpn firefox
Or if you want to move an already running process to the cgroup, well... you can't! That seems to be due to the NAT (masquerade) function:
iptables -nvL -t nat doesn't match when the cgroup is switched, but
iptables -nvL -t mangle does match.
# Get PID of the process (we'll then suppose it's 1234)
# Add to cgroup - THIS DOESN'T WORK! Silently fails to produce the final result.
sudo echo 1234 > /sys/fs/cgroup/net_cls/novpn/tasks
# Remove - but this works...
sudo echo 1234 > /sys/fs/cgroup/net_cls
Credits: No answer were working as expected, but a mix of them did: chripell answer evolware article Per process routing take 2: using cgroups, iptables and policy routing, How do I make a specific process NOT going through a OpenVPN connection?, Kill switch for OpenVPN on the basis of iptables
Related question: iptables and cgroups v2 (netfilter's xt_cgroup)