Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have send a large amount of data from one machine to another. If I send with rsync (or any other method), it will go at a steady 320kb/sec. If I initiate two or three transfers at once, each will go at 320, and if I do four at once, they will max out the link.

I need to be able to send data as fast as possible, so I need a tool that can do inverse multiplexing with file transfers. I need a general solution, so running split on the source machine and catting them together at the other end is not practical. I need this to work in an automated fashion.

Is there a tool that does this, or do I need to make my own? The sender is CentOS, receiver is FreeBSD.

share|improve this question
add comment

11 Answers

Proof it all adds up - I present the 'holy grail' of remote mirror commands. Thanks to davr for the lftp suggestion.

lftp -c "mirror --use-pget-n=10 --verbose sftp://username:password@server.com/directory" 

The above will recursively mirror a remote directory, breaking each file into 10 threads as it transfers!

share|improve this answer
add comment

There are a couple tools that might work.

  • LFTP - supports FTP, HTTP, and SFTP. Supports using multiple connections to download a single file. Assuming you want to transfer a file from remoteServer to localServer, install LFTP on localServer, and run:

    lftp -e 'pget -n 4 sftp://userName@remoteServer.com/some/dir/file.ext'

    The '-n 4' is how many connections to use in parallel.

  • Then there are the many 'download accelerator' tools, but they generally only support HTTP or FTP, which you might not want to have to set up on the remote server. Some examples are Axel, aria2, and ProZilla

share|improve this answer
add comment

How is your data structured? A few large files? A few large directories? You could spawn off multiple instances of rsync on specific branches of your directory tree.

It all depends on how your source data is structured. There are tons of unix tools to slice, dice, and reassemble files.

share|improve this answer
    
Arbitrary data. Sometimes it's a large directory, sometimes a single file. –  ZimmyDubZongyZongDubby Nov 26 '09 at 20:13
add comment

You may be able to tweak your TCP settings to avoid this problem, depending on what's causing the 320KB/s per connection limit. My guess is that it is not explicit per-connection rate limiting by the ISP. There are two likely culprits for the throttling:

  1. Some link between the two machines is saturated and dropping packets.
  2. The TCP windows are saturated because the bandwidth delay product is too large.

In the first case each TCP connection would, effectively, compete equally in standard TCP congestion control. You could also improve this by changing congesting control algorithms or by reducing the amount of backoff.

In the second case you aren't limited by packet loss. Adding extra connections is a crude way of expanding the total window size. If you can manually increase the window sizes the problem will go away. (This might require TCP window scaling if the connection latency is sufficiently high.)

You can tell approximately how large the window needs to be by multiplying the round trip "ping" time by the total speed of the connection. 1280KB/s needs 1280 (1311 for 1024 = 1K) bytes per millisecond of round trip. A 64K buffer will be maxed out at about 50 ms latency, which is fairly typical. A 16K buffer would then saturate around 320KB/s.

share|improve this answer
add comment

If you can setup passwordless ssh login, then this will open 4 concurrent scp connections (-n) with each connection handling 4 files (-L):

find . -type f | xargs -L 4 -n 4 /tmp/scp.sh user@host:path

File /tmp/scp.sh:

#!/bin/bash

#Display the help page
function showHelp()
{
    echo "Usage: $0 <destination> <file1 [file2 ... ]>"
}

#No arguments?
if [ -z "$1" ] || [ -z "$2" ]; then
    showHelp
    exit 1
fi

#Display help?
if [ "$1" = "--help" ] || [ "$1" = "-h" ]; then
    showHelp
    exit 0
fi

#Programs and options
SCP='scp'
SCP_OPTS='-B'
DESTINATION="$1";shift;

#Check other parameters
if [ -z "$DESTINATION" ]; then
    showHelp
    exit 1
fi

echo "$@"

#Run scp in the background with the remaining parameters.
$SCP $SCP_OPTS $@ $DESTINATION &
share|improve this answer
add comment

Try sort all files on inode (find /mydir -type f -print | xargs ls -i | sort -n) and transfer them with for example cpio over ssh. This will max out your disk and make the network you bottleneck. Faster than that it's hard to go when going across network.

share|improve this answer
    
that's downright sneaky :) –  warren Dec 1 '09 at 12:49
    
I can't guarantee that all filesystems get a boost from this, it's depending on how inode layout is done. –  Jimmy Hedman Dec 1 '09 at 13:48
    
The bottleneck is that each TCP connection is limited to 320KB/sec. I want to send files in parallel TCP connections so that I get 320*NumConnections up to the limit of the network (about 1200KB/sec). Sorting by inode doesn't achieve this. –  ZimmyDubZongyZongDubby Dec 1 '09 at 20:35
    
What is limiting the TCP speed? A router between the machines? –  Jimmy Hedman Dec 1 '09 at 21:25
    
My ISP. Net neutrality? HA! –  ZimmyDubZongyZongDubby Dec 1 '09 at 21:35
show 1 more comment

I know a tool that can transfer files in chunks. The tool is called 'rtorrent' package/port that's available on both hosts ;) BitTorrent clients often reserve disk space before the transfer, and chunks are written directly from sockets to the disk. Additionally, you'll be able to review ALL transfers' states in a nice ncurses screen.

You can create simple bash scripts to automate "*.torrent" file creation and ssh a command to the remote machine so it downloads it. This looks a bit ugly, but I don't think you'll find any simple solution without developing :)

share|improve this answer
1  
If only two machines are involved in the file transfer, how can a torrent help? The idea of a torrent is a swarm of seeders making the data available to a client requestor. –  DaveParillo Dec 4 '09 at 7:16
    
You're right. But who said it's not useful with a single seeder? ;) –  kolypto Dec 4 '09 at 20:12
2  
If a torrent client creates multiple TCP connections with a single peer, then this would solve OP's problem. However, I do not know if torrent clients do really create multiple TCP connections with single peers. –  chronos Dec 7 '09 at 12:08
add comment

FTP uses multiple connections for downloads. If you can set up a secure channel for FTP over a VPN or FTP over SSH, you should be able to max out your network link. (Note that special considerations are required for FTP over SSH--see the link.)

FTPS (FTP over SSL) might also do what you need.

You could also use an SFTP client that supports multiple connections, but I'm not sure if SFTP supports multiple connections for a single file. This should do what you need most of the time, but may not give you the maximum throughput when you only have to transfer one large file.

share|improve this answer
    
Wouldn't SFTP be far easier and just as (if not more) secure? –  Mark Renouf Dec 6 '09 at 15:09
    
@rob: where did you get that "FTP uses multiple connections for file transfers" from? Some clients do allow multiple streams for downloading from FTP, but there is definitely no FTP client/server combo allowing multiple streams for uploading to FTP. –  chronos Dec 7 '09 at 12:05
    
@Mark: Yes, SFTP would probably be easier and equally secure, but I don't know if it supports multiple connections for transferring a single file. Thanks for the suggestion though; I'll add it to the list. –  rob Dec 7 '09 at 19:08
    
@chronos: Sorry it wasn't clear; I was suggesting that ZimmyDubZongyZongDubby use FTP to download from the CentOS server to the FreeBSD client. I've updated the answer to specifically say "downloads" instead of "file transfers." –  rob Dec 7 '09 at 19:18
add comment

If you have few and large files use lftp -e 'mirror --parallel=2 --use-pget-m=10 <local_dir> <remote_dir>' <ftp_server>: you'll download 2 files with each splited in 10 segments with a total os 20 ftp connections to <ftp_server>;

If you have a large amount of small files, then use lftp -e 'mirror --parallel=100 <local_dir> <remote_dir>' <ftp_server>: you'll download 100 files in parallel without segmentation, then. A total of 100 connections will be open.

you can use --continue to resume downloads :)

share|improve this answer
add comment

Solution 1: I'm not sure if this is practical in your case, but you could create a spanned archive (for example, a tarfile split into chunks, or a spanned 7zip archive), then use multiple instances of rsync to send them over the network and reassemble/extract them on the other side. You could write a general-purpose script whose arguments are the directory to be transferred and the number of connections to use. The obvious downside is that you'll need twice as much free space on both sides, and will have the additional overhead of archiving/extracting the files on both ends.

Solution 2: a better solution would be to write a script or program that divides the large directory tree into subtrees based on size, then copies those subtrees in parallel. It might simplify things if you copy the entire directory structure (without the files) first.

share|improve this answer
    
Anyone care to elaborate on the downvote? –  rob Dec 7 '09 at 23:44
add comment

Are you two machines running in a trusted environment? You could try netcat. On the server side:

tar -czf - ./yourdir | nc -l 9999

and on the client:

nc your.server.net 9999 > yourdir.tar.gz

You can have the client connection use an ssh tunnel:

ssh -f -L 23333:127.0.0.1:9999 foo@your.server.net sleep 10; \
    nc 127.0.0.1 23333 > yourdir.tar.gz

Even an entire partition can be moved this way:

dd if=/dev/sda1 | gzip -9 | nc -l 9999

and on the client:

nc your.server.net 9999 > mysda1.img.gz

.

Note

netcat is not the most secure transfer tool out there, but in the right environment is can be fast because it has such low overhead.

HowtoForge has a good examples page.

share|improve this answer
    
This seems like a generic answer that doesn't answer his question. I can't see how any of your solutions would transfer in parallel, nc is just a single connection as far as I know –  davr Dec 7 '09 at 20:42
    
You may be right, however, using nc, you have control over the ports opened. You can specify 10,000 if you are so inclined. –  DaveParillo Dec 8 '09 at 5:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.