Two NICs, One Subnet — Linux appears to be primarily transmitting on one NIC, and primarily receiving on the other NIC. How?

Question

I recently pulled some hardware out of storage and put together a fileserver and for whatever reason, the motherboard has two gigabit NICs on it. Since I was tinkering, I figured I might as well plug both of them in and see what happens. My original guess was that Ubuntu would assign one NIC as a primary route with a higher priority, similar to what happens when you connect to both WiFi and Ethernet at the same time.

But alas, what I'm finding is that it primarily seems to be transmitting on one interface and receiving on the other. This is really causing me to scratch my head: I can't figure out how or why it's doing this. My thoughts are that it would make more sense for the server to send and receive data on the same connection, as it would complicate things by sending acknowledgments using a different interface. I've tested this using both iSCSI and NFS with the same results.

ip route:

ashten@mass-storage:~$ ip route
default via 192.168.4.1 dev enp6s0 proto dhcp src 192.168.4.191 metric 100 
default via 192.168.4.1 dev enp5s0 proto dhcp src 192.168.4.57 metric 100 
192.168.4.0/24 dev enp6s0 proto kernel scope link src 192.168.4.191 
192.168.4.0/24 dev enp5s0 proto kernel scope link src 192.168.4.57 
192.168.4.1 dev enp6s0 proto dhcp scope link src 192.168.4.191 metric 100 
192.168.4.1 dev enp5s0 proto dhcp scope link src 192.168.4.57 metric 100

ip -6 route:

::1 dev lo proto kernel metric 256 pref medium
2601:1c1:8804:57b3::/64 dev enp5s0 proto ra metric 100 expires 86394sec pref medium
2601:1c1:8804:57b3::/64 dev enp6s0 proto ra metric 100 expires 86394sec pref medium
fe80::/64 dev enp5s0 proto kernel metric 256 pref medium
fe80::/64 dev enp6s0 proto kernel metric 256 pref medium
default proto ra metric 100 expires 3594sec mtu 1500 
    nexthop via fe80::226d:31ff:fe01:2b8 dev enp6s0 weight 1 
    nexthop via fe80::226d:31ff:fe01:2b8 dev enp5s0 weight 1 pref medium

ifconfig:

ashten@mass-storage:~$ ifconfig
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.4.57  netmask 255.255.255.0  broadcast 192.168.4.255
        inet6 fe80::6ef0:49ff:fe57:3fc  prefixlen 64  scopeid 0x20<link>
        inet6 2601:1c1:8804:57b3:6ef0:49ff:fe57:3fc  prefixlen 64  scopeid 0x0<global>
        ether 6c:f0:49:57:03:fc  txqueuelen 1000  (Ethernet)
        RX packets 34563916  bytes 49179525874 (49.1 GB)
        RX errors 0  dropped 128937  overruns 0  frame 0
        TX packets 138348330  bytes 30171509363 (30.1 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp6s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.4.191  netmask 255.255.255.0  broadcast 192.168.4.255
        inet6 fe80::6ef0:49ff:fe57:3ec  prefixlen 64  scopeid 0x20<link>
        inet6 2601:1c1:8804:57b3:6ef0:49ff:fe57:3ec  prefixlen 64  scopeid 0x0<global>
        ether 6c:f0:49:57:03:ec  txqueuelen 1000  (Ethernet)
        RX packets 401203508  bytes 604740226354 (604.7 GB)
        RX errors 0  dropped 121190  overruns 0  frame 0
        TX packets 109130  bytes 99996128 (99.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Mostly I've been pushing files to the server thusfar, so the TX bytes is a bit low on both of them. But notice that on enp6s0, the RX bytes is 600GB, but the TX bytes is under 100MB. Using bmon I can observe the same thing: With only one concurrent file transfer, the server is receiving on enp6s0 while simultaneously transmitting on enp5s0. bmon also reveals that it's IPv6 packets that're being sent and received. Does IPv6 have rules regarding multiple IP addresses that's allowing for this? If so, I'm struggling to find said rule. Or am I totally off-base with my research?

Answer 1

My original guess was that Ubuntu would assign one NIC as a primary route with a higher priority, similar to what happens when you connect to both WiFi and Ethernet at the same time.

Yes, that is generally what I expect to happen as well (for IPv4). I am not 100% sure why the kernel even allows you to have multiple IPv4 default routes with identical 'metric' value, and I don't know how it chooses the "best" route in this situation.

However, IPv6 on Linux is slightly different: identical routes are allowed, and they actually get automatically merged into a "multipath" route with two gateways – the kernel load-balances across both interfaces per-connection.

• Note #1: This is Linux-specific. Other operating systems (Windows, BSD, etc) do not support multipath routes and will just choose one preferred interface for both IPv4 and IPv6, exactly as you would expect.

• Note #2: Linux does support IPv4 multipath routes as well, but they have to be manually defined. The automatic merging that you're seeing is a quirk that has to do with how IPv6 multipath routes were implemented originally (or rather, weren't).

But alas, what I'm finding is that it primarily seems to be transmitting on one interface and receiving on the other. This is really causing me to scratch my head: I can't figure out how or why it's doing this.

The machine doesn't "pull" packets to receive them through a specific interface – the decision is a bit more indirect than that:

0. Your host has sent something from 192.168.4.57, and another device wants to send a reply back.

1. The sender wants to resolve 192.168.4.57 to a MAC address and broadcasts an ARP query.

2. Your host receives the ARP query twice, on both interfaces.

3. Linux by default uses the "weak host" model for IPv4, so it checks whether the requested IP address belongs to the machine as a whole, not whether the address belongs to the specific interface.

4. Your host responds to both queries, indicating that 192.168.4.57 is at <some MAC address>. That is, it responds for both of its MAC addresses.

   (Your switch also learns the source MAC addresses and associates them to the corresponding physical ports.)

5. The sender receives both ARP responses, and (I may be wrong about this) both are processed in order, so a later response overrides an earlier one. As a result, the sender "knows" that 192.168.4.57 is at 6C:F0:49:57:03:EC.

6. When packets from the sender reach your Ethernet switch, it looks at its MAC table to determine the correct output port – because the packets are addressed to 6C:F0:49:57:03:EC, they exit through the switch port that's connected to your enp6s0.

So in order to "tightly" associate each IP address to its corresponding interface, you need to fix step 3 – there is actually sysctl which tells Linux to only respond to ARP queries if they arrive on the correct interface, just as it already does for IPv6. Set this through sysctl.d:

net.ipv4.conf.all.arp_ignore = 1

Does IPv6 have rules regarding multiple IP addresses that's allowing for this? If so, I'm struggling to find said rule.

As mentioned above, the Linux kernel is a bit stricter when handling IPv6 NDP queries than it is for IPv4 ARP queries, and will only respond if the desired address belongs to the same interface as the query. (I believe this is part of the "strong host" model.)

So when the sender tries to resolve 2601:1c1:8804:57b3:6ef0:49ff:fe57:3fc to a MAC address, your host will only respond from the enp5s0 interface.

There may be other factors, but this is the main reason that I can think of.

My thoughts are that it would make more sense for the server to send and receive data on the same connection, as it would complicate things by sending acknowledgments using a different interface.

It does complicate things in some ways, but as long as IP does its job delivering the packets from A to B, it'll keep working and the TCP layer won't even notice the difference. Asymmetric routes are relatively common for Internet traffic – e.g. depending on how well-connected your ISP is, you might be sending an HTTP request via Cogent and receiving the response via Telia...

(What would really complicate things is if packets belonging to the same connection were being sent over both interfaces at once, as this could result in the recipient receiving them out-of-order which will cause problems for TCP. But when you use multipath routes or link aggregation, the OS will try to avoid this by mapping each TCP connection to a single interface.)

---

Side note: You should look into link aggregation (aka "bonding" or "teaming"). Linux supports LACP and a few types of static configurations – this would give you a single bond0 interface which has just one IP address and one MAC address but uses both Ethernet connections simultaneously.

Unlike multipath routes (which only help in one direction), link aggregation will combine the bandwidth of both links when sending and when receiving, though most modes still have the limitation that a single TCP stream will remain limited to a single Ethernet connection.

Aggregation has to be supported on both ends of the Ethernet cable – if the Ethernet connections go into a switch, you need to activate aggregation on that switch.

Answer 2

Fundamental networking rule: If you attach a machine more than one time to some subnet, funny things will start to happen. Don't do that unless you know about these things and have plans to mitigate them.

My original guess was that Ubuntu would assign one NIC as a primary route with a higher priority,

It can't know that both LAN ports go to the same router, so it doesn't. The mechanism works as it would when you'd attach both ports to different LAN segments (which is the normal use case).

similar to what happens when you connect to both WiFi and Ethernet at the same time.

That's a hack, because it can distinguish between WLAN and LAN.

But alas, what I'm finding is that it primarily seems to be transmitting on one interface and receiving on the other.

When you have two interfaces with different IP addresses for the same subnet, outgoing connections basically choose one randomly. That you see more packets on interface than on the other is probably also some random effect, like some kind of service that sends out a lot of packets having bound to one address by chance.

It also should receive packets as an answer on the interface that has the source address only. I don't know why you see more received packets on interface; again, it could be caused by a random effect that connections that send few packets but receive a lot of responses happen to pick the other interface at random.

You can easily check all that with tcpdump or wireshark on both interfaces, and then open a connection with curl or something and see what happens. Answers to the outgoing connection should only come in on the same interface.

Just looking at the statistics is not going to help to find out what actually happens.