91

I want to transfer files (a music folder) between two Linux computers. After searching for the best way to do this, I've seen that there are lots of ways of doing this. I know this has been asked a lot, everywhere and all the time. The main problem with this is that there is no clear, recent consensus on one best way to do this task in 2011 for Linux beginners (even depending on some parameters).

So in the spirit of the Stack Exchange websites, I want this not to be related to my particular situation, but more of a guide to others as well on how to transfer files between two Linux computers over a local network. I think a wiki would be useful for many.

Here's what I found so far:

  • ssh
  • sshfs
  • scp
  • sftp
  • nfs
  • samba
  • giver

What is the easiest? Most flexible? Simplest? Best solution? What are the pros and cons of each? Are there other (better) options? What are the parameters in choosing the best method (solution might depend on number of files, filesize, easiness vs. flexibility, ...)?

2
  • 2
    Could anyone explain where rsync comes to play in all of this?
    – Konerak
    Aug 22 '11 at 8:39
  • jonallard, please don't add the answers to the question (doesn't really make sense to do this, does it?) -- if you feel some answers need additional info, you can suggest edits on them, or create your own answer which summarizes everything!
    – slhck
    Aug 22 '11 at 10:19

11 Answers 11

77

In a Linux environment, for both security and ease of use, ssh is the best way to go. SSH, SSHFS, SCP, and SFTP as you list are all just different services built on top of the SSH protocol. SCP is very easy to use, it works just like CP but you can provide user and machine names in the path. So, we might do a CP like cp ~/music/ ~/newmusic/, but we could just as easily do scp ~/music/ user@host:~/newmusic to send it to the computer named host. That's it - we don't need to set anything up. You'll be prompted for the account password on the other machine if you don't have certificate or some other authentication set up (scp shares those settings with ssh, of course).

SFTP is a tool that makes it easy to do a lot of operations on a remote file system - it works just like FTP, but it runs through SSH so it's secure and requires only an SSH server. man sftp will tell you all about how to use it. I don't use SFTP just to move a folder between two machines, it's more useful when you have a lot of operations to do, like if you're rearranging files on another computer.

SSHFS just extends SFTP in to a file system: it allows you to mount a virtual host in to your file system, so the network stuff happens totally transparently. SSHFS is for semi-permanent setups, not just a one-time file transfer. It takes some more effort to get set up, which you can read about on the project website.

If you need to work in a mixed-OS environment, Samba becomes your next best bet. Windows and OS X support Samba completely automatically, and Linux does as well although it's sometimes rough to use.

7
  • 3
    Exactly the kind of answer I was wishing for: complete, exhaustive, detailed, to the point.
    – jonallard
    Aug 22 '11 at 5:12
  • 2
    One thing though, for scp to work, do we need to set up some kind of ssh server, listener or unblock something on the other side? I'm getting "Connection refused" errors.
    – jonallard
    Aug 22 '11 at 5:28
  • 2
    scp uses ssh, so it will work if SSH works. This means, of course, that you need to have an SSH server running (default in every linux distro I'm aware of) and a connection must be possible (firewalls, NAT, etc must have the appropriate exceptions). Aug 22 '11 at 5:42
  • 8
    Apparently openssh-server has to be installed in Ubuntu Natty.
    – jonallard
    Aug 22 '11 at 5:51
  • 3
    Note that ssh uses encryption, which will cause some additional overhead. If the computers involved have fairly slow CPUs, this might make a difference. In that case netcat or similar (see Caspar's answer) might be preferrable. Of course only if you don't actually need encryption (in a protected LAN).
    – sleske
    Aug 22 '11 at 7:25
69

My personal favorite for cases where security doesn't matter is netcat + tar:

To send a directory, cd to inside the directory whose contents you want to send on the computer doing the sending and do:

tar -cz . | nc -q 10 -l -p 45454

On the computer receiving the contents, cd to where you want the contents to appear and do:

nc -w 10 $REMOTE_HOST 45454 | tar -xz

Replace $REMOTE_HOST with ip / hostname of computer doing the sending. You can also use a different port instead of 45454.

What's actually happening here is that the 'receiving' computer is connecting to the sending computer on port 45454 and receiving the tar'd and gzip'd contents of the directory, and is passing that directly to tar (and gzip) to extract it into the current directory.

Quick example (using localhost as a remote host)

Computer 1

caspar@jumpy:~/nctest/a/mydir$ ls
file_a.txt  file_b.log
caspar@jumpy:~/nctest/a/mydir$ tar -cz . | nc -q 10 -l -p 45454

Computer 2

caspar@jumpy:~/nctest/b$ ls
caspar@jumpy:~/nctest/b$ nc -w 10 localhost 45454 | tar -xz
caspar@jumpy:~/nctest/b$ ls
file_a.txt  file_b.log
10
  • kinda reminds me of sendnet and recnet from Novel's ipx services of the olden days... Aug 22 '11 at 12:33
  • 4
    ## netcat + bzip2 may be a little faster on slow connections ## Sending server # cat file.txt | bzip2 -c | nc -l 1234 ## Receiving server # nc $sending_ip 1234 | bzip2 -cd > file.txt
    – shantanuo
    Aug 24 '11 at 3:20
  • @Caspar: How about editing this answer to suggest bzip2 or lzma compression?
    – einpoklum
    Nov 23 '13 at 18:59
  • 5
    the -q option indicates that you are using openbsd-netcat, while gnu-netcat is also quite common (default in Arch Linux). Could you extend your answer to include the syntax of gnu-netcat?
    – Sebastian
    Jan 6 '14 at 20:48
  • 1
    The current nc man says about the -l option: "It is an error to use this option in conjunction with the -p, -s, or -z options", but strangely it doesn't throw an error when using it. I think using 'nc -l 45454' should work just as well.
    – Claudiu
    Sep 25 '14 at 7:58
19

For one time moves, scp is recommended.

But if you find that this dir may work and you need to move it many times to keep the other position updated then you can use rsync (with ssh).

Since rsync has a lot of arguments I usually put it in a small shell so I get it right (every time). The idea is to only send things that has changed since the last time it ran.

#!/bin/bash

user="nisse"
host="192.168.0.33"

echo "Sync: /home/media/music/"
rsync --archive --delete -v --progress -e "ssh -l $user " /home/media/music/ $host:/home/media/music/

This will move a dir called "/home/media/music/" from the local computer to the the pc called 192.168.0.33, using user "nisse". And delete anything on the target that doesn't exist on the local pc.

2
  • 1
    + for rsync, which seems a tad faster and is very nice if you have to sync the directory later on Aug 23 '11 at 14:55
  • This seems very promising (and easily reuseable) but I've got dirs and files with spaces and I'm getting rsync error: syntax or usage error (code 1) at main.c(1348) [sender=3.1.1] -- any suggestions? Aug 22 '15 at 21:54
13

I would recommend you trying alternatives instead going straight with SSH for moving files inside your own LAN as the overhead is IMMENSE. I would go with Caspar's solution if this one for whatever reason won't work for you:

At the source:

$ python3 -m http.server {PICK_YOUR_PORT}

On destination:

$ wget -r {ip / hostname}:{port}/{File / Directory}

This won't only be lighter than using SSH but far faster with speeds ranging 45~65MiB on standard CAT6 UTP.
If you really want to squeeze the most out of the connection try replacing wget with lftp and using pget -n20 and mirror -r commands.

9

The fastest is probably netcat (as caspar described).

I like the combination of tar & ssh, which is secure and still fast:

On the source

tar -cf - . | ( ssh user@target && cd /target/path && tar -xf - )

Doing that as root, it preserves file permissions. Or use -p on both sides. Also -S might be considered if you have sparse files.

It's possible to reduce the encryption overhead of ssh if you use arcfour as cipher which works with openSSH:

tar -cpSf - . | ( ssh -c arcfour user@targethost && cd /target/path && tar -xpSf - )

To update the remote path, rsync is perfect:

rsync -av --sparse --delete -e "ssh -c arcfour" . root@targethost:/target/path
1
  • 1
    FWIW, I recently did a transfer test between two directly connected modern laptops using rsync both with the arcfour option and no specific -e argument. I noticed no difference in speeds. Dec 24 '17 at 16:15
5

If it absolutely has to be done over the LAN, I'd use rsync, as it will pick up where it left off if it gets interrupted. It also has a few other tricks for minimizing the amount of data that gets transferred, although I doubt that many/any of them would be relevant to the case of copying a music library to a virgin location. If security is a concern, just set RSYNC_RSH=ssh first and the data will be tunneled over ssh.

If I were actually doing it, though, I probably wouldn't use the LAN at all. I'd copy the files onto, then off of, a USB hard drive. In my experience, this can easily be multiple orders of magnitude faster than going over the LAN, despite having to copy the files twice - USB 2.0 is rated for 480Mbps, which is faster than anything short of gigabit ethernet, plus it's less sensitive to conditions which will degrade the performance of a LAN. It's also completely OS-independent, provided you use a file system that all involved machines can handle - I'd recommend VFAT/FAT32, since that's pretty much universal.

2
  • 1
    I'm a fan of the (so-called) sneakernet as well, but it's worth pointing out that while USB 2 is supposed to be able to get 480Mbps, I've only ever seen it get 30MB/s (about 240Mbps).. maybe I just have cheap USB<->SATA hardware ;) Also, FAT32 is pretty much universal but is a no-go for copying things like DVD images due to the 4gb filesize constraint; it's worth pointing out so people don't get too frustrated by the "out of space" error message which Windows (at least) gives.
    – Caspar
    Aug 23 '11 at 0:55
  • @Caspar: Good caveats! Thanks for mentioning them; I always forget about the FAT32 file size limit... Aug 23 '11 at 9:09
2

I would suggest rsync as it will copy files incrementally. You can set it up to copy either only modified or new files only once you have done the initial update. You can use ssh as a transport layer if you wish.

2

I use Unison, which is an awesome file synchronizer over many different protocols. You can configure it to use scp, rcp, ftp or even locally on the filesystem between two folders. I use it to synchronize my music library, as it can transfer multiple files at once over the network and is really tuneable in its configuration. I keep my music collection backed up and in sync over 2-3 computers. It will only copy changed files, and does so by keeping an index on both ends of the transfer so as to be able to tell when a client has changed the file or when the server file has changed.

Your mileage may vary, but it's certainly a lot better than scping your entire music collection every time you add a new song :)

2

A lot of great answers were mentioned, I thought I would add another alternative, rclone and is a personal favourite because of it's support for many cloud storage providers and support for Windows.

On the host machine:

rclone serve sftp . --addr 0.0.0.0:2222

This will spin up an SFTP server, listening on port 2222 on all addresses. By default it will use the current user's authorized keys file (~/.ssh/authorized_keys)

On the client system:

You can use any SFTP-supported GUI or CLI program. Unfortunately, it doesn't seem to support SCP. But you can connect to the SFTP server in the terminal similar to

sftp -P 2222 dave@10.75.0.200

The rclone serve command also supports additional protocols, including Webdav, DLNA, FTP, HTTP(s), if you're limited to a certain protocol.

If you need to synchronize two folders, you could use the built-in sync command. However you'll need to configure rclone, using rclone config and add the remote FTP/SFTP server.

An example:

rclone sync /home/dave/projects localSftpServer:projects
0

I followed ssh process for passwordless login first http://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps/

For scripts and text files the following works for me just fine

To transfer data from local host to remote host. cat localfile | ssh <user>@<ip> "cat > <path>/<remotefile>"

To transfer data from remote host to local host. ssh <user>@<ip> "cat > <path>/<remotefile>" | cat > localfile

This works for me to transfer files on embedded systems that don't have ssh client or scp built in.

No scp - only ssh.

0

Caspar's netcat approach is awesome for its simplicity, but there may be situations where you want don't want to send unencrypted files over the network.

To really get this right, you should probably use a mature solution like scp, SFTP, samba, or magic-wormhole (a really cool one, but as of the time of writing unfortunately not great for sending huge folders).

But considering you are in this situation:

  • you don't want to setup a user account for the other party, or don't know how to configure it so the other person can't execute shell commands on your PC (so SCP/SFTP is out of the picture)
  • the configuration overhead of FTPS/samba is too much as well for a one-off transfer
  • your folders are too large for magic-wormhole
  • and you don't expect serious attackers

The following solution may be interesting:

Using gpg -c for encryption

Pull mode:

serve (host):     tar -cz . | gpg -c --cipher-algo AES256 | ncat -lp 8000
fetch (client):   ncat HOST 8000 --recv-only | gpg -d | tar -xz

(here, using nmap's ncat implementation. For others, see below..)

Push mode:

In case the receiver is easier to connect to than the sender, you can easily invert their roles by changing the netcat commands in the pipeline:

send (client):    tar -cz . | gpg -c --cipher-algo AES256 | ncat HOST 8000
recv (host):      ncat -lp 8000 --recv-only | gpg -d | tar -xz

The host command always has to be executed first.

Password via command line:

If you want, the password can be provided on both ends using

  • gpg [...] --batch --passphrase-fd 3 3<<<'PASSWORD', or
  • gpg [...] --batch --passphrase-file <(cat <<<'PASSWORD'), or simply
  • gpg [...] --batch --passphrase PASSWORD. However, this form should mostly be avoided, especially on multi-user systems.

Caution:

As far as I understand, GPG does authenticate the ciphertext against the password, but this happens very late, so that the tar extraction process may already be impossible to roll back and could have overwritten files in your local directory if some part of the ciphertext was replaced or incorrectly transmitted. For this reason, always cd to a new empty directory before doing this.

Note that I'm no expert on GPG/crypto and can't promise this is not immensively unsafe in other ways as well.

Things you could also try

OpenBSD netcat

I think it works, but no guarantees:

Pull mode:

serve (host):     ENCRYPT | nc -lNp 8000
fetch (client):   nc -d HOST 8000 | DECRYPT

Push mode:

send (client):    ENCRYPT | nc -N HOST 8000
receive (host):   nc -ldp 8000 | DECRYPT

GNU netcat

In theory, this could work:

Pull mode:

serve (host):     ENCRYPT | nc -clp 8000
fetch (client):   nc HOST 8000 | DECRYPT

Push mode:

send (client):    ENCRYPT | nc -c HOST 8000
receive (host):   nc -lp 8000 | DECRYPT

However, in my trials it often failed due to closing the connection before all bytes were sent/received.

This can be mitigated by either not specifying the -c option (in which case you have to close the connection manually using Ctrl+C, when you think it's done), or by building in some delay on the sender side:

(ENCRYPT; sleep 10s) | nc -clp 8000

Different netcat implementations on client/server

Should work.

Encrypt the stream using openssl enc

Serve files:

tar -cz . | openssl enc -e -aes-256-ctr -pbkdf2 | ncat -lp 8000

Fetch files:

ncat HOST 8000 --recv-only | openssl enc -d -aes-256-ctr -pbkdf2 | tar -xz

openssl will ask you for the password. Alternatively, you can provide the password using the -k PASSWORD command-line option on both ends, or on multi-user systems, using -pass file:<(cat<<<PASSWORD).

Beware that this does not provide authenticated encryption - which means that there is no guarantee that the stream is unaltered. So, someone could really mess up your filesystem if they change a few bytes. Although, you get basic redundancy checks in the gzip archive enabled by the -z option, this will not reliably protect you. This is probably fine to protect a love letter against your friends, but don't use it if there may be someone even more nefarious in the network!

Unfortunately, openssl's enc doesn't support any authenticated encryption modes.

Using zip for encryption

Server:

tar -cz . | zip -e -Ppassword | ncat -lp 8000

Client:

ncat HOST 8000 --recv-only | funzip -password | tar -xz

Seems to work, but prints the following error:

funzip error: invalid compressed data--length error

(not due to network errors, I get the same error when calling funzip in a local pipe, like echo text | zip | funzip.)

Not the answer you're looking for? Browse other questions tagged or ask your own question.