1

I have a number of NICs in my Centos7 box, eno1, eno2 and enp2s0f0, enp2s0f1, enp2s0f2, enp2s0f3. The only one connected with a ethernet cable is eno2, however, I am able to see all except eno1 from a different machine (Win10) using arp -a, they show up with the same MAC as eno2. I can ssh to all the 192.168.<3-6>.82 no problem.

brctl show tells me there are no bridges, and cat /proc/sys/net/ipv4/ip_forward gives me 0. What gives?

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ac:1f:6b:d6:a9:f4 brd ff:ff:ff:ff:ff:ff
    inet 10.10.1.163/24 brd 10.10.1.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:fed6:a9f4/64 scope link
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ac:1f:6b:d6:a9:f5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.71/16 brd 192.168.255.255 scope global noprefixroute dynamic eno2
       valid_lft 86263sec preferred_lft 86263sec
    inet6 fe80::2482:20ed:7ae1:9b00/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
4: enp2s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 3c:fd:fe:ea:53:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.82/24 brd 192.168.3.255 scope global enp2s0f0
       valid_lft forever preferred_lft forever
5: enp2s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 3c:fd:fe:ea:53:11 brd ff:ff:ff:ff:ff:ff
    inet 192.168.4.82/24 brd 192.168.4.255 scope global enp2s0f1
       valid_lft forever preferred_lft forever
6: enp2s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 3c:fd:fe:ea:53:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.5.82/24 brd 192.168.5.255 scope global enp2s0f2
       valid_lft forever preferred_lft forever
7: enp2s0f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 3c:fd:fe:ea:53:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.82/24 brd 192.168.6.255 scope global enp2s0f3
       valid_lft forever preferred_lft forever
3
  1. The first issue is with the client system, because it shouldn't be making those ARP queries in the first place.

    Since you see wrong-subnet addresses in arp -a, that indicates your client system has routes added that define all those /24's as "on-link".

    (Perhaps you've assigned unnecessary addresses to its own Ethernet interace? Or perhaps you accidentally configured it as a /16?)

  2. The second issue is with the server system responding to those ARP queries if they arrive on the "wrong" interface.

    It's actually default behavior in Linux's IPv4 (I believe it's part of the "weak host" model) – it treats IPv4 addresses as if they belong to the entire host rather than a specific interface. You can disable it by setting this sysctl on the multi-NIC machine:

    net.ipv4.conf.all.arp_ignore = 1
    
  3. The third issue is with the server system selecting the "wrong" interface when it sends TCP/IP responses.

    Again, this is default behavior (part of the same weak-host model) – the multi-NIC machine will always respond using the same default route regardless of the source IP that it's responding from, because the IPv4 routing table only lets you match on 'to', not 'from'.

    To solve this problem for IPv4 you'd need policy routing via ip rule. (However, the IPv6 routing table on Linux can match on both source and destination, which allows much simpler solutions.) Some other operating systems also make connections "stick" to their original interface, which can be achieved with iptables marks and the same ip rule

  • i only added that as a sidenote or clue to what's going on. there seems to be some sort of forwarding that i'm unaware of. i can still SSH to each of the <3-6>.82 addresses directly without those ports being physically connected. – Piotr Feb 14 at 18:30
  • 1
    Since you see those addresses in arp -a in the first place, that indicates your client has routes added that define all those /24's as "on-link" for the same interface. And as part of the same weak-host model, the multi-NIC machine will always respond through its existing default route regardless of the source IP that it's responding from. – user1686 Feb 14 at 18:48
  • Aha, I see what you mean now. I was unaware of the strong/weak model and how they worked. Thanks! – Piotr Feb 14 at 21:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.