I'm trying to understand a counter-intuitive phenomenon where I get only ~ 400 Mbit/s through a gigabit connection when the CPU is idle, but ~ 850 Mbit/s when the CPU is under full load (stress test utility). I suspect this is a symptom of some misconfiguration, in the end I would like to be able to saturate the link regardless of the CPU usage. The question is: what could cause the poor network performance, and why does it depend on the CPU usage in this strange way?
I'm glad for any hints, thanks for your time!
Systems and hardware
- Lynx SENYO Mini PC (GE5 PCF55) running up-to-date Arch Linux kernel version 5.0.8 (called the client for reference).
NIC is a Realtek RTL8168:
# lspci -nn | grep Ethernet 08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 03)Connection: CAT 6 patch cable <-> CAT 7 in-wall cable <-> CAT 6 patch field <-> HPE OfficeConnect Switch <-> CAT 6 patch cable <-> supermicro home server (called the server for reference). Another PC gets ~ 950 Mbit/s on the exact same connection to the server with the exact same cables, so I guess this isolates the problem to the client.
Test setup
On the server:
# iperf3 -D -s --bind 192.168.128.1On the client while CPUs are idle:
# iperf3 -c 192.168.128.1 Connecting to host 192.168.128.1, port 5201 [ 5] local 192.168.128.100 port 60738 connected to 192.168.128.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 39.8 MBytes 333 Mbits/sec 0 310 KBytes [ 5] 1.00-2.00 sec 40.5 MBytes 339 Mbits/sec 0 322 KBytes [ 5] 2.00-3.00 sec 40.3 MBytes 338 Mbits/sec 0 322 KBytes [ 5] 3.00-4.00 sec 41.0 MBytes 344 Mbits/sec 0 322 KBytes [ 5] 4.00-5.00 sec 36.9 MBytes 310 Mbits/sec 0 322 KBytes [ 5] 5.00-6.00 sec 53.1 MBytes 445 Mbits/sec 0 322 KBytes [ 5] 6.00-7.00 sec 53.7 MBytes 450 Mbits/sec 0 322 KBytes [ 5] 7.00-8.00 sec 54.7 MBytes 459 Mbits/sec 0 338 KBytes [ 5] 8.00-9.00 sec 54.0 MBytes 453 Mbits/sec 0 338 KBytes [ 5] 9.00-10.00 sec 54.0 MBytes 453 Mbits/sec 0 338 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 468 MBytes 393 Mbits/sec 0 sender [ 5] 0.00-10.04 sec 467 MBytes 390 Mbits/sec receiverOn the client while running
stress -c 4(all four CPUs busy on 100%)# iperf3 -c 192.168.128.1 Connecting to host 192.168.128.1, port 5201 [ 5] local 192.168.128.100 port 60742 connected to 192.168.128.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 102 MBytes 854 Mbits/sec 0 356 KBytes [ 5] 1.00-2.00 sec 101 MBytes 845 Mbits/sec 0 375 KBytes [ 5] 2.00-3.00 sec 101 MBytes 846 Mbits/sec 0 392 KBytes [ 5] 3.00-4.00 sec 101 MBytes 844 Mbits/sec 0 409 KBytes [ 5] 4.00-5.00 sec 100 MBytes 843 Mbits/sec 0 409 KBytes [ 5] 5.00-6.00 sec 101 MBytes 843 Mbits/sec 0 409 KBytes [ 5] 6.00-7.00 sec 101 MBytes 845 Mbits/sec 0 409 KBytes [ 5] 7.00-8.00 sec 100 MBytes 841 Mbits/sec 0 451 KBytes [ 5] 8.00-9.00 sec 101 MBytes 843 Mbits/sec 0 451 KBytes [ 5] 9.00-10.00 sec 101 MBytes 851 Mbits/sec 0 510 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1008 MBytes 846 Mbits/sec 0 sender [ 5] 0.00-10.04 sec 1005 MBytes 840 Mbits/sec receiver
Unsuccessful attempts
- I thought maybe its a frequency scaling issue, but
cpupower frequency-set -g performancemakes no difference. - Reading about issues with my specific NIC, I tried both drivers, the one present in the kernel (r8169) and the one from Realtek (r8168). The general picture is the same, the numbers are a bit different. With the Realtek driver I get 556 Mbits/s (idle CPU) vs 788 Mbits/s (busy CPU).
- Cross-checked a live Ubuntu 18.04 LTS (kernel 4.18.0) with the same results.
- Running
irqbalancemakes no difference. - Disabling IPv6 makes no difference.
Additional information (glad to add more on request)
Output from ip
# ip addr show dev enp8s0
2: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:14:0b:81:3f:8a brd ff:ff:ff:ff:ff:ff
inet 192.168.128.100/24 brd 192.168.128.255 scope global noprefixroute enp8s0
valid_lft forever preferred_lft forever
Output from ethtool:
# ethtool enp8s0
Settings for enp8s0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000033 (51)
drv probe ifdown ifup
Link detected: yes
# ethtool -i enp8s0
driver: r8169
version:
firmware-version: rtl_nic/rtl8168d-1.fw
expansion-rom-version:
bus-info: 0000:08:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
Interrupts:
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
...
26: 145689 0 0 394154 PCI-MSI 4194304-edge enp8s0
...
# cat /proc/softirqs
CPU0 CPU1 CPU2 CPU3
HI: 1 42 46 1
TIMER: 11550 32469 10688 11494
NET_TX: 85221 4 7 189091
NET_RX: 145692 0 14 394372
BLOCK: 21 450 39 22765
IRQ_POLL: 0 0 0 0
TASKLET: 136092 93 97 384278
SCHED: 10554 29735 8720 9726
HRTIMER: 0 0 0 0
RCU: 6154 8303 5838 6506
