11

This link describes how to copy tarred files to minimise the amount of date sent over the network. I am trying to do something slightly different.

I have a number of remote files on different subdirectory levels:

remote:/directory/subdir1/file1.ext
remote:/directory/subdir1/subsubdir11/file11.ext
remote:/directory/subdir2/subsubdir21/file21.ext

And I have a file that lists all of them:

remote:/directory/allfiles.txt

To copy them most efficiently, on the remote site I could just do

tar zcvf allfiles.tgz `cat /directory/allfiles.txt`

but there is not enough space to do that.

I do have enough storage space on my local disc. Is there a way to tar an incoming stream from a remote server (using scp or ssh for the transfer)?

Something like /localdir$ tar zc - | ssh remote `cat /directory/allfiles.txt` I would guess - but that would only list the remote files on the local host.

4 Answers 4

8

You got it almost right, just run the tar on the remote host instead of locally. The command should look something like the following:

ssh remote_host tar cvfz - -T /directory/allfiles.txt > remote_files.tar.gz
3
  • Excellent, that's what I was hoping to get! I had thought about this but expected the redirection to be interpreted on the remote machine not the local. I guess the hyphen takes care of that?
    – alle_meije
    Jul 6, 2015 at 7:41
  • I don't think that the -v is a good idea. You're going to use extra bandwidth for that feature. You could instead verify the results with a tar tvf remote_files.tar.gz once the transfer is done. Dec 27, 2019 at 19:32
  • If you really have to have to most minimal bits transfered over the network, sure. But really how much more bandwidth is '-v' going to take for showing the file progress? Data send over with ssh is probably also compressed too.
    – dirk
    Dec 28, 2019 at 21:38
1

If you have enough space on your local disk and your goal is to minimise the amount of date sent over the network maybe it is enough to enable the compression with scp or rsync :

scp -avrC remotehost:/path/to/files/file1 /files/file2 ...  local/destination/path

Of course you can do a little script to loop over each file and do an scp compressed transfer, even without the use of tar. All is more cosy with rsync

rsync -avz --files-from=FILE remotehost:/path/to/files  local/destination/path

You can connect via ssh to the remote host and write there

tar cvzf - -T list_of_filenames | ssh Local_Hostname tar xzf -

References:

  • from man scp:

    -C      Compression enable.  Passes the -C flag to ssh(1) to enable compression.
    
  • from man rsync

    --files-from=FILE       read list of source-file names from FILE
    -z, --compress          compress file data during the transfer
    --compress-level=NUM    explicitly set compression level
    
3
  • Thanks for mentioning the --files-from option for rsync, that is handy! The compression option of scp is efficient for network traffic -- would the script you mention mean typing the password many times (or storing it in ~ /.ssh)?
    – alle_meije
    Jul 6, 2015 at 7:47
  • rsync can be even better of scp (check the options -S for sparse files), moreover it requires the authentication only once, and you can skip files just present. For the scp scripted solution you really will want to use the autmate SSH authentication.
    – Hastur
    Jul 6, 2015 at 9:18
  • You may want to fix "date" with "data" in "[...] the amount of dat[e] sent [...]" — and I think that's the main reason to transform that data to a compressed tarball in the first place... Dec 27, 2019 at 19:28
1

If we take your start command:

tar zcvf allfiles.tgz `cat /directory/allfiles.txt`

then all we need to do is make sure that the file ends up remotely. So lets tar to standard out, pipe that though netcat sand save it as a local file.

Alternatively, something like this:

local: nc -l 1234 | gunzip | tar zcvf allfiles.tgz
remote: cat /directory/allfiles.txt | gzip | nc localIP 1234

Explanation:

Local, step 1: Input from netcat. -l to listen on some port. Send the output to whereever you want it.
Remote: step 1: youyr already exiting list of file. gzipped for minimum network data. Send results to local.
local: Step 2: Uhm, we are already getting gzipped files. Lets uncompress them before feeding them to tar.

0

In case you wanted to compress the data as much as possible to reduce the amount of bandwidth used up, you can run the tar command with the xz tool like so:

ssh host 'tar cvf - -T /directory/allfiles.txt | xz -9' > files.tar.xz

Notice the extra single quotes ('). These are required because the pipe needs to be run by the remote shell, not your local shell.

Here are a few drawbacks to be aware of, though:

  1. the process uses a lot of memory (around 700Mb) so you want to make sure your server has that much free memory available (preferably around 1Gb or more) to not start swapping
  2. The xz command is very slow when using -9. If you do not pay extra for using the CPU for hours (in case you are transferring Gb of data...) then you should be fine.
  3. It also uses only one single CPU, which may be a good thing here. In any event, you're going to have one CPU pegged at 100% usage for the duration.

Still, if your bandwidth is the most expensive part of your service, that's a good idea to use the best possible compression.

Note that the compression will slow down the speed of download. If you have enough room on your server disk drive, then you can try to create the tarball on the server and once done with that process, do one fast transfer. I mention that because if your connection is a bit flaky, the ssh command above may break midway and then you'd have to restart the whole process... If that is a real problem, using rsync would certainly be your best bet. The advantage with rsync is that it sends files one at a time and it's capable of restarting where it left off. It also compresses the data, but as far as I know, it only knows of gzip (but it has been a while since I checked, they may now support better compressors).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .