2

I'm trying to transfer files as fast as possible across a 10 gig network connection. Up until now, I've been using rsync to move large files across the network, but I'm trying mbuffer now because I've heard its faster. The trouble is that I don't get as large an increase in performance as I'm expecting, especially since I'm using a 10 gig network connection between the two computers.

These are my current mbuffer arguments:

mbuffer -m 1G -s 512M -P 10 -i "$currentFile" | ssh $outputTarget "cat - > $outputDir/$filename" > $debugOut

Does anybody have any tips for how I can improve speeds over the network? I'm new to mbuffer and don't really know which arguments would work best for my situation. Right now I'm getting about 110 MiB/s for a 1 GB file I made out of random numbers.

3
  • 1
    What are you reading the files off of? If it's just one hard drive, it's possible that you've reached the maximum sequential read speed for the disk. Jul 22, 2015 at 15:04
  • It is just one hard drive. That's entirely possible, yeah. On the other hand, I thought mbuffer buffered to RAM, and the buffer fills instantly before writing.
    – Inglonias
    Jul 22, 2015 at 15:05
  • see also unix.stackexchange.com/questions/48399/…
    – rogerdpack
    Sep 24, 2018 at 16:22

1 Answer 1

3

This answer to probably far too late to help the OP, so the following is for posterity.

The bottleneck likely has nothing to do with mbuffer but with the fact that data is being sent through an ssh-encryped connection. In other words, the bottleneck is the speed at which the single-threaded ssh process can encrypt traffic on the sending host and decrypt it on the receiving host.

For your purpose, it's probably better to use mbuffer's -O option on the sender and the -I option on the receiver. That makes mbuffer send the data directly across an unencrypted TCP pipe, which will push your network as hard as the OS, drivers, and hardware will allow.

It's a two-step process because you need to run mbuffer on both the sending host and the receiving host, like this:

  1. First, run mbuffer on the receiver. It will listen for a connection on the port given by the -I option (5567 is just an example; choose your own port number):

    ssh $outputTarget mbuffer -I 5567 \>$outputDir/$filename
    
  2. Then initiate the transfer on the sender:

    mbuffer -i "$currentFile" -O $outputTarget:5567
    

As outlined above, the receiving mbuffer process running on $outputTarget will listen for and accept a TCP connection from anyone who connects to that socket, not just from the sending mbuffer process that is started in step 2. So you have to be mindful of the fact that mbuffer, when used in this way, is not as secure as ssh, not only because there's no encryption of the data, but also because there's no way to be certain that the intended sender is the only one initiating a connection to the receiver's listen port. Unless someone is port-scanning your receiver or otherwise knows what you are doing and is trying to exploit you, the latter caveat is rarely problematic. The lack of encryption, on the other hand, might very well be a show-stopper depending on the use case.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .