Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a folder with tens of thousands of files that I need to scp to another box. As doing one file at a time is very timeconsuming, and as I don't have harddrive space to make a compressed tar archive of all the tiles — I'd like to do the following:

Tar whole archive on the fly and pass it directly over scp to the other box and untar it immediately there. That way I can get full bandwidth (no need to start each file) and I won't run out of space by duplicating the whole archive.

My A box is OSX, B is Ubuntu.

Is there a one-liner for this?

share|improve this question
Why does it have to be a one liner? Couldn't you just run the three commands, i.e. create a tar, send it, and untar? – slhck Jul 16 '13 at 20:33
Why use scp when ssh can stream: tar c | ssh tar x? Clearly add whatever options you need so that the tree is from where you need to where you want it. And that is a one-liner. – Dan D. Jul 16 '13 at 20:34
@slhck I don't have space for a duplicate of my files. – knutole Jul 16 '13 at 20:41
@PauloAlmeida it's not the file-size that is the issue; there are overheads for each file sent over networks (scp specifically creates a separate transfer for each individual file), and tarring them into a single file reduces that overhead. – evilsoup Jul 16 '13 at 22:24
With the answer below, I get a steady 2MB/s bandwidth, with scp I get a start-n-stop for each file, averageing at .75-1MBM/s for each file, but with hundreds of milliseconds break in between files. No competiton. – knutole Jul 16 '13 at 22:50
up vote 3 down vote accepted

scp works only on files that are already on the drive. Since you don't want to create the file (tar) before sending it, you have to use ssh and tar directly:

tar -c -f - ./path/to/compress | ssh eee.lan tar -C /where/to/extract -x -f-
share|improve this answer
Sweet! I love a straight answer! Thanks! – knutole Jul 16 '13 at 22:39
Bare hyggelig :) – Ярослав Рахматуллин Jul 16 '13 at 22:49
Btw, I douche! ;) Takk! – knutole Jul 16 '13 at 22:50
I tried this without -f (just tar -c ./path/to/compress | ssh eee.lan tar -x) and it worked. Are there any advantages, corner cases or portability issues that make -f necessary? – Paulo Almeida Jul 17 '13 at 0:50

I would recommend using rsync; in case your connection gets interrupted rsync can pick up where it left (with a little overhead). rsync can working over ssh directly so it will still be secure.

rsync -av -e ssh /path/to/send rsync://

share|improve this answer
Will this put everything into a single file so that I won't get the overhead for each file sent? – knutole Jul 16 '13 at 22:36
Rsync uses a datastream, not a file. But it's only a single datastream, yes. The real saver is in case your transfer gets interrupted or only a few files have changed and you still want to sync both ends: rsync compares directories and/or files on both ends and only sends the different ones. The tar/scp solution above will transfer everything, every time. – JvO Jul 16 '13 at 23:06
OK, thanks for that! I was only moving one big batch this time (and still am!), but sync/compare is awesome so glad to know it. – knutole Jul 18 '13 at 0:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .