0

I have two machines locally connected via 10 Gigabit Ethernet. System Information:

  • Machine A: Intel Core i7 8700K, 32 GB DDR4 memory, Intel X550-T2 10GbE NIC, Arch Linux
  • Machine B: AMD EPYC 3101, 256 GB DDR4 memory, ASRock Rack EPYC3101D4I-2T onboard 10GbE NIC, Debian 11

When transferring large files, speeds are lower than expected. I have tried several protocols (SMB: 250 MB/s, SCP: 430 MB/s, HTTP: 630 MB/s, FTP: 800 MB/s), but none of them get close to line speed, at least not in both directions.

I have already ruled out various bottlenecks.

  • iperf3 shows 9.4 Gb/s (1120 MB/s) in both directions
  • files are copied from and to tmpfs, with the lowest read/write speed being 2.8 GB/s
  • system monitor shows network rates similar to transfer speeds, ruling out network overhead

I suspect that the main bottleneck is the slower CPU in machine B, with different protocols causing different CPU loads and thus resulting in different transfer speeds.

Taking a closer look at FTP (inetutils ftp client, vsftpd server), I get the following transfer speeds and CPU loads (where X <- Y means client X is downloading from server Y, and X -> Y means client X is uploading to server Y):

  • A <- B: 1120 MB/s (A: 60% CPU, B: 30% CPU)
  • A -> B: 809 MB/s (A: 30% CPU, B: 100% CPU)
  • B <- A: 704 MB/s (B: 100% CPU, A: 10% CPU)
  • B -> A: 1120 MB/s (B: 75% CPU, A: 30% CPU)

Active vs. Passive mode made no difference in my tests.

It seems that receiving causes more CPU load than sending, and the FTP client experiences more CPU load than the FTP server. This way, I get close to line speed when transferring from B -> A, but not the other way around.

How can I transfer files in both directions without being bottlenecked by the CPU?

5
  • Please limit your post to a single question. Also, please note that networking is a multi-purpose system, designed to handle a wide range of abilities and functions. It is very difficult over a normal network link to get "line speed": there will always be overhead. I'm not saying that's the only cause of what you're seeing, but having correct expectations is a helpful thing. Mar 9, 2022 at 4:20
  • @music2myear That is okay. I would consider the 1120 MB/s close enough to line speed. I am only concerned about the other direction, where I only get 700-800 MB/s. Regarding the single question ideal, how should I ask about multiple aspects of my situation? Would duplicating the entire description into multiple posts really be sensible, especially considering the relatedness of the sub-questions?
    – Jules
    Mar 9, 2022 at 4:29
  • 1
    If the situation warrants it, perhaps. But rather, start with a single question. This is all pretty clearly laid out in the Help section you were walked through when you joined Su. Mar 9, 2022 at 4:52
  • 1
    You might want to give NFS a try. I have a ultra-low end microserver where the switch from FTP to NFS made a huge difference in CPU load and improved the transfer speed quite a bit. Mar 9, 2022 at 5:00
  • @music2myear I edited my question.
    – Jules
    Mar 9, 2022 at 5:21

0

You must log in to answer this question.

Browse other questions tagged .