I have two machines locally connected via 10 Gigabit Ethernet. System Information:
- Machine A: Intel Core i7 8700K, 32 GB DDR4 memory, Intel X550-T2 10GbE NIC, Arch Linux
- Machine B: AMD EPYC 3101, 256 GB DDR4 memory, ASRock Rack EPYC3101D4I-2T onboard 10GbE NIC, Debian 11
When transferring large files, speeds are lower than expected. I have tried several protocols (SMB: 250 MB/s, SCP: 430 MB/s, HTTP: 630 MB/s, FTP: 800 MB/s), but none of them get close to line speed, at least not in both directions.
I have already ruled out various bottlenecks.
- iperf3 shows 9.4 Gb/s (1120 MB/s) in both directions
- files are copied from and to tmpfs, with the lowest read/write speed being 2.8 GB/s
- system monitor shows network rates similar to transfer speeds, ruling out network overhead
I suspect that the main bottleneck is the slower CPU in machine B, with different protocols causing different CPU loads and thus resulting in different transfer speeds.
Taking a closer look at FTP (inetutils ftp client, vsftpd server), I get the following transfer speeds and CPU loads (where X <- Y means client X is downloading from server Y, and X -> Y means client X is uploading to server Y):
- A <- B: 1120 MB/s (A: 60% CPU, B: 30% CPU)
- A -> B: 809 MB/s (A: 30% CPU, B: 100% CPU)
- B <- A: 704 MB/s (B: 100% CPU, A: 10% CPU)
- B -> A: 1120 MB/s (B: 75% CPU, A: 30% CPU)
Active vs. Passive mode made no difference in my tests.
It seems that receiving causes more CPU load than sending, and the FTP client experiences more CPU load than the FTP server. This way, I get close to line speed when transferring from B -> A, but not the other way around.
How can I transfer files in both directions without being bottlenecked by the CPU?