I have a folder with tens of thousands of files that I need to scp to another box. As doing one file at a time is very timeconsuming, and as I don't have harddrive space to make a compressed tar archive of all the tiles — I'd like to do the following:

Tar whole archive on the fly and pass it directly over scp to the other box and untar it immediately there. That way I can get full bandwidth (no need to start each file) and I won't run out of space by duplicating the whole archive.

My A box is OSX, B is Ubuntu.

Is there a one-liner for this?

  • Why does it have to be a one liner? Couldn't you just run the three commands, i.e. create a tar, send it, and untar? – slhck Jul 16 '13 at 20:33
  • 2
    Why use scp when ssh can stream: tar c | ssh tar x? Clearly add whatever options you need so that the tree is from where you need to where you want it. And that is a one-liner. – Dan D. Jul 16 '13 at 20:34
  • @slhck I don't have space for a duplicate of my files. – knutole Jul 16 '13 at 20:41
  • 1
    @PauloAlmeida it's not the file-size that is the issue; there are overheads for each file sent over networks (scp specifically creates a separate transfer for each individual file), and tarring them into a single file reduces that overhead. – evilsoup Jul 16 '13 at 22:24
  • 1
    With the answer below, I get a steady 2MB/s bandwidth, with scp I get a start-n-stop for each file, averageing at .75-1MBM/s for each file, but with hundreds of milliseconds break in between files. No competiton. – knutole Jul 16 '13 at 22:50

scp works only on files that are already on the drive. Since you don't want to create the file (tar) before sending it, you have to use ssh and tar directly:

tar -c -f - ./path/to/compress | ssh eee.lan tar -C /where/to/extract -x -f-
  • 1
    Bare hyggelig :) – Ярослав Рахматуллин Jul 16 '13 at 22:49
  • Btw, I douche! ;) Takk! – knutole Jul 16 '13 at 22:50
  • 1
    I tried this without -f (just tar -c ./path/to/compress | ssh eee.lan tar -x) and it worked. Are there any advantages, corner cases or portability issues that make -f necessary? – Paulo Almeida Jul 17 '13 at 0:50

I would recommend using rsync; in case your connection gets interrupted rsync can pick up where it left (with a little overhead). rsync can working over ssh directly so it will still be secure.

rsync -av -e ssh /path/to/send rsync://user@ubuntu.box/path/to/receive

  • Will this put everything into a single file so that I won't get the overhead for each file sent? – knutole Jul 16 '13 at 22:36
  • 2
    Rsync uses a datastream, not a file. But it's only a single datastream, yes. The real saver is in case your transfer gets interrupted or only a few files have changed and you still want to sync both ends: rsync compares directories and/or files on both ends and only sends the different ones. The tar/scp solution above will transfer everything, every time. – JvO Jul 16 '13 at 23:06
  • OK, thanks for that! I was only moving one big batch this time (and still am!), but sync/compare is awesome so glad to know it. – knutole Jul 18 '13 at 0:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.