1

This is my home network which I am learning heaps on.

Please see detailed relevant details of VLAN and relevant zone firewall rules at the bottom of post as its lengthy

I have successfully assigned 2 NIC's (Management/DMZ) to each of the 6 windows servers successfully and use my Windows laptop in LAN to access things behind management for RDP/Web access for switch/router etc. I can ping each server from my Windows LAN machine to their DMZ and Management IP's and get a return.

So I have added a Ubuntu 16 LTS server few days ago with same config but can not ping both IP's from the Windows laptop.

Here is my /etc/network/interfaces file

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.67.100
        netmask 255.255.255.0
#       network 192.168.67.0
#       broadcast 192.168.67.255
        gateway 192.168.67.253
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 192.168.67.253
        dns-search on.fake.network

# Management network interface
auto eth1
iface eth1 inet static
        address 192.168.7.100
        netmask 255.255.255.0
#       network 192.168.7.0
        broadcast 192.168.7.255

#persistent static routes
up route add -net 192.168.1.0/24 gw 192.168.7.253 dev eth1

My IP route table

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.67.253  0.0.0.0         UG    0      0        0 eth0
192.168.1.0     192.168.7.253   255.255.255.0   UG    0      0        0 eth1
192.168.7.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.67.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0

So with this config I can ping 192.168.7.100 but not 192.168.67.100 I take out the persistent static route and its opposite :/

How can I win, this is fairly easy in Windows, but has my stumped in Ubuntu

* ADDITIONAL DETAILS *

Ubiqiti ERL3 router creates VLANS

VLAN 7 - Management (192.168.7.0) - Router interface (GW) IP is 192.168.7.253
VLAN 13 - LAN (192.168.1.0) - Router interface (GW) IP is 192.168.1.253
VLAN 67 - DMZ (192.168.67.0) - Router interface (GW) IP is 192.168.67.253

Windows laptop on LAN connection

VLAN 13 - LAN (192.168.1.15)

Ubuntu Server (Through Hyper-V host) has 2 NIC's

eth0 - VLAN 67 - DMZ (192.168.67.100)
eth1 - VLAN 7 - Management (192.168.7.100)

Ping is initiated from LAN 192.168.1.15 to both DMZ/MGMT 192.168.67.100/192.168.7.100

Here are my condensed firewall rules that are relevant to this scenario, removed extras like SSH/HTTP access etc etc.

Noting "address-group mgmtfromlan" contains several IP's from LAN including 192.168.1.15 (The laptop in the LAN VLAN)

name lan-dmz {
    default-action drop
    enable-default-log
    rule 1 {
        action accept
        state {
            established enable
            related enable
        }
    }
    rule 2 {
        action drop
        log enable
        state {
            invalid enable
        }
    }
    rule 100 {
        action accept
        description "Allow ICMP"
        log enable
        protocol icmp
    }
}
name lan-mgmt {
    default-action drop
    enable-default-log
    rule 1 {
        action accept
        state {
            established enable
            related enable
        }
    }
    rule 2 {
        action drop
        log enable
        state {
            invalid enable
        }
    }
    rule 100 {
        action accept
        description "Allow ICMP"
        log enable
        protocol icmp
        source {
            group {
                address-group mgmtfromlan
            }
        }
    }
}
name dmz-lan {
    default-action drop
    enable-default-log
    rule 1 {
        action accept
        state {
            established enable
            related enable
        }
    }
    rule 2 {
        action drop
        log enable
        state {
            invalid enable
        }
    }
}
name mgmt-lan {
    default-action drop
    enable-default-log
    rule 1 {
        action accept
        state {
            established enable
            related enable
        }
    }
    rule 2 {
        action drop
        log enable
        state {
            invalid enable
        }
    }
    rule 100 {
        action accept
        description "Allow ICMP"
        log enable
        protocol icmp
    }
}

* ROUTING TABLE ON WINDOWS LAPTOP 192.168.1.15 *

===========================================================================
Interface List
 15...b8 ca 3a d4 bb bc ......Intel(R) 82579LM Gigabit Network Connection #2
  5...3c a9 f4 03 73 ed ......Microsoft Wi-Fi Direct Virtual Adapter #2
 19...00 ff d4 0e 47 e9 ......TAP-Windows Adapter V9
  7...3c a9 f4 03 73 ec ......Intel(R) Centrino(R) Ultimate-N 6300 AGN #2
  1...........................Software Loopback Interface 1
 12...00 00 00 00 00 00 00 e0 Microsoft Teredo Tunneling Adapter
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0    192.168.1.253     192.168.1.15    291
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    331
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    331
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    331
      192.168.1.0    255.255.255.0         On-link      192.168.1.15    291
     192.168.1.15  255.255.255.255         On-link      192.168.1.15    291
    192.168.1.255  255.255.255.255         On-link      192.168.1.15    291
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    331
        224.0.0.0        240.0.0.0         On-link      192.168.1.15    291
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    331
  255.255.255.255  255.255.255.255         On-link      192.168.1.15    291
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
          0.0.0.0          0.0.0.0    192.168.1.253  Default
          0.0.0.0          0.0.0.0    192.168.1.253  Default
===========================================================================

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    331 ::1/128                  On-link
  1    331 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

Thanks in advance

7
  • 1
    Not enough information. You give your settings, that don't work, so they can't be assumed to be reliable to know your topology. You should have an explicit description of all your topology, for eg with ascii art, including when relevant both ips of each node involved. Example of ping -v results, arp -an results (what is doing the ping?)... Could be for example asymetric routing with reverse path filtering issues or anything else
    – A.B
    Oct 21 '17 at 11:19
  • I have expanded the original post with a lot more info I hope. Included more information about the firewalls on the Ubiquiti relevant to this question and clarified everything as much as possible. Thanks Oct 21 '17 at 12:19
  • What is the routing table of 192.168.1.15 ? Is 192.168.67.253 and 192.168.7.253 the same router which is ERL3 and also has for example the 192.168.1.253 ip? or are there other routers not seen? A picture would help best, because else I'd have to transform mentally your configuration (from a router I don't know) into this picture. I don't think you have firewall issues since you can get both working separately. I think it's related to asymmetric routing. ping/ping reply received on one interface when expected on the other or alike. But this could happen on the router or on the ubuntu box.
    – A.B
    Oct 21 '17 at 12:25
  • If you did a few tcpdump-like captures (on the router, the only one that can see everything) that would really help. Ideally multiple apture, one per interface, knowing if it's "in" or "out".
    – A.B
    Oct 21 '17 at 12:30
  • Added routing table to bottom of the original post. All gateways ie .67.253, 7.253 and 1.253 are coming from the same router Ubiquiti ERL3. I am also not suspecting the firewall, suspecting my inexperience with Ubuntu/Linux in this regard. I think next step is to figure out a detailed log of the router. All firewall rules accept/deny/drop etc are logged currently. Thanks Oct 21 '17 at 12:45
0

With the default route set, the ubuntu VM receives a packet on eth1 (from 192.168.1.15: ping 192.168.7.100). It will either:

  • drop the packet because of reverse path filtering: packet received on an interface where there's no route to the source of this packet.
  • answer through eth0 because 192.168.1.15 matches the default route via 192.168.1.253 on eth0

Behaviour depends on /proc/sys/net/ipv4/conf/{all,eth1}/rp_filter values, but drop is the default: all/rp_filter = 1 and eth1/rp_filter = 1

In the case where it's not dropped: router ERL3 sees a reply packet from 192.168.1.7.100 which should be in lan-mgmt, but transits from lan-dmz. Depending on settings perhaps described in your router's configuration you wrote, but that I don't know:

  • will drop it, more or less for the same reason ubuntu drops it when rp_filter is enabled: the route for it is elsewhere
  • will accept it anyway.

Let's consider that the ERL3 accepts it. Then the "fix" is simple: lower security on ubuntu by disabling reverse path filtering:

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter

ref: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt (rp_filter)

(the way it's working, eth0 stays protected, but that's irrelevant here)

Try it first, if it works, problem solved. Consider replacing both echo 0 with echo 2 for eth1 only instead (loose mode). But consider that's still an issue: asymmetric routing can create other problems especially with firewalling.

If the router ERL3 is filtering too (ie: above didn't work): ubuntu has to be aware of 192.168.1.0/24 's existence for both interfaces and use table and rule settings for a correct routing. That's a bit difficult to explain all the why in addition to how, here's a reference, that isn't exactly for this case but can be mostly based upon (which I did here):

ref: http://lartc.org/howto/lartc.rpdb.multiple-links.html

I chose arbitrary tables 7 and 67 to match 192.168.7.0/24 and 192.168.67.0/24. Start from empty routes (beside the local lan routes automatically created) and:

ip route add 192.168.67.0/24 dev eth0 src 192.168.67.100 table 67
ip route add default via 192.168.67.253 table 67
ip route add 192.168.7.0/24 dev eth1 src 192.168.7.100 table 7
ip route add 192.168.1.0/24 via 192.168.7.253 table 7 #enable only LAN to have a route to access MGMT. Feel free to replace 192.168.1.0/24 with default, you have a firewall too.

I don't know if this is really required, but that's in the documentation so...:

ip route add 192.168.7.0/24 dev eth1 src 192.168.7.100
ip route add 192.168.67.0/24 dev eth0 src 192.168.67.100

Your default route stays the same, here again in ip route 's syntax:

ip route add default via 192.168.67.253

And the really important part which tells to use the right interface:

ip rule add from 192.168.67.100 table 67
ip rule add from 192.168.7.100 table 7

Final note: make sure ubuntu doesn't route, else a compromised DMZ host might attempt to manage an other through ubuntu:

echo 0 > /proc/sys/net/ipv4/conf/default/forwarding
echo 0 > /proc/sys/net/ipv4/conf/all/forwarding
3
  • fixed main default route (67 not 7)
    – A.B
    Oct 21 '17 at 14:15
  • Thanks for the very detailed information. I am going to be attempting both answers shortly I hope, but very interesting way it works :) Oct 22 '17 at 10:45
  • Thanks A.B this worked great! The first part did not work by itself the echo 0 parts, traffic was still not flowing through. Doing the subsequent section with the ip route add's worked right away. After I completed the test from both answers I realize they are very similar/same as the thomas-krenn link I tried first for whatever reason. Appreciate your help yesterday on narrowing down my problem and responding with the right answer first. Appreciated! Oct 22 '17 at 12:46
0

Your static route overrides the default gateway.

When the route exists all traffic to 192.168.1.x leaves through eth1 using 192.168.7.253 as the gateway.

When the route does not exist all traffic to 192.168.1.x leaves through eth0 using 192.168.67.253 as the gateway.

The path from your pc to the server is ok, but the return path takes a different route back, depending on which interface you pinged. This is called asymmetric routing and is typically not allowed by security devices. It is probably blocked on your firewall. In essence, the return traffic coming through a different route is "unrelated" and unrecognized traffic, so it is dropped.

I think the easiest way to resolve this issue is to insure traffic takes the correct route back depending on which interface received the data. That can be done by setting up a default gateway on both interfaces.

Unfortunately this isn't as easy as it sounds. You'll need to create alternate routing tables and routing rules.

See if these write ups will help: https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System

https://askubuntu.com/questions/310355/networking-with-multiple-nics

2
  • Thanks for the links and information. I spent a considerable amount of time before asking the question here only to find the thomas-krenn link via google during conversations with A.B. Very interesting way and will reply after I get a chance to test both methods provided in both answers. Oct 22 '17 at 10:48
  • It took no time at all to edit the config as per the information from thomas-krenn. IT works, I can now ping both interfaces on the Ubuntu Server and should be able to play with different ports to different interfaces. My firewall is blocking rules by default that are not established or related, so if ping came in on one interface and went out the other this should be blocked by my ubiquiti firewall/working as designed. Oct 22 '17 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.