Originally posted at Unix and Linux but nobody was able to answer it, so m migrating the question here:
My question is regarding Source Based Policy Routing on CentOS 5 with 2 WANs plus a LAN (NAT) port with Load Balancing, but first before anything some remarks before starting to describe the problem...
I know this topic has been brought many times here at stack exchange and seems that the top 5 answers are (ordered from the most to the least):
- Disable rp_filter
- Mark/Connmark based policy routing
- IP Based policy routing (add more IPs)
- Install pfSense, Shorewall, Ubuntu?, Etc...
- Buy expensive Cisco/3com/Juniper/Etc... Router
Most of the times, some of this answers are correct but for me solutions 1 & 2 haven't workout (i don't discard at least point 2 cause i may have some issues with my setup), point 3 is basically isolating a problem rather than solving it (also adds complexity to the routing tables) and solutions 4 & 5 are just out of the scope since i don't have resources to buy specialized hardware nor can take offline the server since it is on production so to summarize replacing the CentOS server with something "better" is off the table.
Ok now back in the problem, lets first describe the current setup...
eth1: IP: 10.0.0.1, GW: 10.0.0.1, NM: 255.255.255.0 (LAN) eth0: IP: 10.0.1.1, GW: 10.0.1.254, NM: 255.255.255.0 (ISP1 - ADSL Router) eth2: IP: 10.0.2.1, GW: 10.0.2.254, NM: 255.255.255.0 (ISP2 - ADSL Router)
# Controls IP packet forwarding net.ipv4.ip_forward = 1 # Controls source route verification net.ipv4.conf.default.rp_filter = 0 # Do not accept source routing net.ipv4.conf.default.accept_source_route = 0 # Controls the use of TCP syncookies net.ipv4.tcp_syncookies = 1 # Ignoring broadcasts request net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.icmp_ignore_bogus_error_messages = 1
# # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep 2 ISP1 3 ISP2
10.0.1.0/24 dev eth0 src 10.0.1.1 table ISP1 default via 10.0.1.254 dev eth0 table ISP1
10.0.2.0/24 dev eth2 src 10.0.2.1 table ISP2 default via 10.0.2.254 dev eth2 table ISP2
fwmark 2 table ISP1 from 10.0.1.1 table ISP1
fwmark 3 table ISP2 from 10.0.2.1 table ISP2
*filter :INPUT DROP [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] # Basic Rules -A INPUT -i lo -j ACCEPT -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT # SSH -A INPUT -i eth0 -m tcp -p tcp --dport 22 -j ACCEPT -A INPUT -i eth2 -m tcp -p tcp --dport 22 -j ACCEPT # OpenVPN -A INPUT -i eth0 -m udp -p udp --dport 1194 -j ACCEPT -A INPUT -i eth2 -m udp -p udp --dport 1194 -j ACCEPT # Allow everything from LAN -A INPUT -i eth1 -j ACCEPT # Allow everything from the VPN -A INPUT -i tun0 -j ACCEPT # Default Drop on everything else -A INPUT -j DROP # Allow forwarding from LAN and VPN -A FORWARD -i eth1 -j ACCEPT -A FORWARD -i tun0 -j ACCEPT # Allow all outbound traffic -A OUTPUT -o lo -j ACCEPT -A OUTPUT -o eth1 -j ACCEPT COMMIT *nat :PREROUTING ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :OUTPUT ACCEPT [0:0] # DNAT to Developer Box (SSH Server) -A PREROUTING -i eth0 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.0.0.200:2222 -A PREROUTING -i eth2 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.0.0.200:2222 # SNAT -A POSTROUTING -o eth0 -j SNAT --to-source 10.0.1.1 -A POSTROUTING -o eth2 -j SNAT --to-source 10.0.2.1 COMMIT *mangle :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] # Mark Based Routing? (based on NerdBoys site) -A PREROUTING -j CONNMARK --restore-mark -A PREROUTING --match mark --mark 2 -j ACCEPT -A PREROUTING -i eth0 -j MARK --set-mark 2 -A PREROUTING --match mark --mark 3 -j ACCEPT -A PREROUTING -i eth2 -j MARK --set-mark 3 -A PREROUTING -j CONNMARK --save-mark COMMIT
Load Balancing is being possible with the gwping bash script which basically monitors the 2 wans (eth0 and eth2) and set up the default routes and weights in the server like this (while in load balance or the 2 wans up and running):
ip route replace default scope global nexthop via 10.0.1.1 dev eth0 weight 1 nexthop via 10.0.2.1 dev eth1 weight 1
The problem i have is that even with this setup that lot of people seems to agree is the correct one, m still having issues with accessing services inside the network from the outside (specifically the ssh developer box and the OpenVPN one) even that the packets are being "marked" and routed accordingly the answer from the dev box goes always to the wrong path. I don't know if m missing something in the mangle or nat area or m misunderstanding source based routing at all, anyway if someone know how to make this work accordingly it will be kindly appreciated.
My sources for this setup are:
lartc.org/lartc.html#LARTC.RPDB.MULTIPLE-LINKS fatalsite.net/?p=90 nerdboys.com/2006/05/05/conning-the-mark-multiwan-connections-using-iptables-mark-connmark-and-iproute2/ policyrouting.org/PolicyRoutingBook/ONLINE/CH08.web.html unix.stackexchange.com/questions/58635/iptables-set-mark-route-diferent-ports-through-different-interfaces unix.stackexchange.com/questions/22770/two-interfaces-two-addresses-two-gateways bulma.net/body.phtml?nIdNoticia=2145
PS1: I found a website that its saying that the marks in the routing table should be + 1 different from the iptables marks (kim.attr.ee/2010/08/source-based-policy-routing-on-centos.html) is this true? or this website is super-incorrect.
Update 15/08/2013 22:15
After more researching and debugging, i found a website which says that i forgot to add the SNAT part at the post-routing table so i add this rules to the iptables config:
-A POSTROUTING --match mark --mark 2 -j SNAT --to-source 10.0.1.1 -A POSTROUTING --match mark --mark 3 -j SNAT --to-source 10.0.2.1
But m still unable to connect to the devbox from the outside the network. On the good side an a iptables -t nat -nvL POSTROUTING gives a hint about the workings of connmark based policy routing, so maybe its something related at the ISP1 and ISP2 router edge:
Chain POSTROUTING (policy ACCEPT 520 packets, 56738 bytes) pkts bytes target prot opt in out source destination 0 0 SNAT all -- * * 0.0.0.0/0 0.0.0.0/0 MARK match 0x2 to:10.0.1.1 6 312 SNAT all -- * * 0.0.0.0/0 0.0.0.0/0 MARK match 0x3 to:10.0.2.1 903 70490 SNAT all -- * eth0 0.0.0.0/0 0.0.0.0/0 to:10.0.1.1 931 78070 SNAT all -- * eth2 0.0.0.0/0 0.0.0.0/0 to:10.0.2.1
Also i add more info from my setup, please somebody throw me a life-saver since m starting to ran out of ideas... >.<
ip route show:
10.8.0.2 dev tun0 proto kernel scope link src 10.8.0.1 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.1 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.1 10.8.0.0/24 via 10.8.0.2 dev tun0 10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.1 169.254.0.0/16 dev eth2 scope link default nexthop via 10.0.1.254 dev eth0 weight 1 nexthop via 10.0.2.254 dev eth2 weight 1
ip rule show:
0: from all lookup 255 1024: from all fwmark 0x2 lookup ISP1 1025: from all fwmark 0x3 lookup ISP2 2024: from 10.0.1.1 lookup ISP1 2025: from 10.0.2.1 lookup ISP2 32766: from all lookup main 32767: from all lookup default