3

I have a functional ospf adjacency between a server with 2 10Gb NICs and the L3 switch it is connected to over two /30 networks. A /32 IP address is assigned to the lo interface which is pingable and passes traffic reliably.

However in testing the ecmp via tcpdump, I either see traffic pin to one interface or duplicated across both, depending on the destination. More importantly, the sibling server which is configured similarly, but with another loopback ip address, traffic is pinned to one NIC.

The routes in the route table, particularly the peer server, all show equally weighted interfaces, yet the desired result of achieving >10Gbps throughput between each server is not met.

At the least, I would like to see terrible <10Gbps performance due to CPU overhead, but with alternate routes utilizing both uplinks.

I'm trying to solve the issue of drbd being bound to 1 IP/NIC with only a single 10Gbps link to sync between systems, while having >10Gbps of potential iSCSI initiator requests. The limiting factor here being the throughput of the peer link.

default via 10.0.4.1 dev enp9s0f0 proto zebra metric 1

10.0.4.0/30 dev enp9s0f0 proto kernel scope link src 10.0.4.31 metric 107
10.0.4.4/30 dev enp9s0f1 proto kernel scope link src 10.0.4.31 metric 106
10.0.4.8/30 via 10.0.4.1 dev enp9s0f0 proto zebra metric 14
10.0.4.12/30 via 10.0.4.5 dev enp9s0f1 proto zebra metric 14
10.0.4.30 proto zebra metric 24
        nexthop via 10.0.4.5 dev enp9s0f1 weight 1
        nexthop via 10.0.4.1 dev enp9s0f0 weight 1
Improve this question

1 Answer 1

Reset to default
3

As far as I know, both ECMP and LACP try to pin each flow (connection) to a single path – I believe it's to avoid TCP problems caused by reordering.

In Linux ECMP, this can be adjusted using sysctl net.ipv4.fib_multipath_hash_policy to take into account either just the IP address pair (L3 hashing) or additionally the TCP/UDP port pair (L3+4 hashing).

Changing the sysctl value to 1 would distribute multiple flows/connections across both routes.

If that is still not suitable, some types of L2 bonding/teaming might work.

Improve this answer
2
  • Thank you for getting me in the right direction. My network hardware doesn't support VXLAN, which I understand abstracts the L3 network into a single L2 domain (simply put). And one of the benefits it's lauded for is the ability to provide true load balancing of L2 traffic since each Frame is re-encapsulated in a L3 packet which can be transparently balanced. Am I misunderstanding the feature as able to aggregate multiple L2 links into a single abstracted L2 link ontop of the L3 fabric, and the benefits are still grounded in the limitations of LACP bonds?
    – ACiD GRiM
    Jun 11, 2018 at 18:24
  • I'm trying to work around the lack of VXLAN support in my configuration. I'll select this answer once I've been able to test the multipath hash policies.
    – ACiD GRiM
    Jun 11, 2018 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.