2

I have a folder with tens of thousands of files that I need to scp to another box. As doing one file at a time is very timeconsuming, and as I don't have harddrive space to make a compressed tar archive of all the tiles — I'd like to do the following:

Tar whole archive on the fly and pass it directly over scp to the other box and untar it immediately there. That way I can get full bandwidth (no need to start each file) and I won't run out of space by duplicating the whole archive.

My A box is OSX, B is Ubuntu.

Is there a one-liner for this?

9
  • Why does it have to be a one liner? Couldn't you just run the three commands, i.e. create a tar, send it, and untar?
    – slhck
    Jul 16, 2013 at 20:33
  • 2
    Why use scp when ssh can stream: tar c | ssh tar x? Clearly add whatever options you need so that the tree is from where you need to where you want it. And that is a one-liner.
    – Dan D.
    Jul 16, 2013 at 20:34
  • @slhck I don't have space for a duplicate of my files.
    – knutole
    Jul 16, 2013 at 20:41
  • 1
    @PauloAlmeida it's not the file-size that is the issue; there are overheads for each file sent over networks (scp specifically creates a separate transfer for each individual file), and tarring them into a single file reduces that overhead.
    – evilsoup
    Jul 16, 2013 at 22:24
  • 1
    With the answer below, I get a steady 2MB/s bandwidth, with scp I get a start-n-stop for each file, averageing at .75-1MBM/s for each file, but with hundreds of milliseconds break in between files. No competiton.
    – knutole
    Jul 16, 2013 at 22:50

2 Answers 2

4

scp works only on files that are already on the drive. Since you don't want to create the file (tar) before sending it, you have to use ssh and tar directly:

tar -c -f - ./path/to/compress | ssh eee.lan tar -C /where/to/extract -x -f-
3
  • 1
    Bare hyggelig :) Jul 16, 2013 at 22:49
  • Btw, I douche! ;) Takk!
    – knutole
    Jul 16, 2013 at 22:50
  • 1
    I tried this without -f (just tar -c ./path/to/compress | ssh eee.lan tar -x) and it worked. Are there any advantages, corner cases or portability issues that make -f necessary? Jul 17, 2013 at 0:50
5

I would recommend using rsync; in case your connection gets interrupted rsync can pick up where it left (with a little overhead). rsync can working over ssh directly so it will still be secure.

rsync -av -e ssh /path/to/send rsync://[email protected]/path/to/receive

3
  • Will this put everything into a single file so that I won't get the overhead for each file sent?
    – knutole
    Jul 16, 2013 at 22:36
  • 2
    Rsync uses a datastream, not a file. But it's only a single datastream, yes. The real saver is in case your transfer gets interrupted or only a few files have changed and you still want to sync both ends: rsync compares directories and/or files on both ends and only sends the different ones. The tar/scp solution above will transfer everything, every time.
    – JvO
    Jul 16, 2013 at 23:06
  • OK, thanks for that! I was only moving one big batch this time (and still am!), but sync/compare is awesome so glad to know it.
    – knutole
    Jul 18, 2013 at 0:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .