I have to transport a lot of files from one PC to another (both Linux). I would like to use scp for that, but scp only allows for transferring one file at a time.

How can I do this?

I have

  • No possibility to use rsync or any other protocol
  • No possibility to use passphrase-free certificates (but have a certificate with a passphrase)
  • A list of files to transfer and a list with the destination path of the files on the other server
  • The files are spread out over a lot of directories, and not all the files in the directories I want to copy

If possible, I would like to gzip and ungzip transparently to save bandwidth!

Is this possible?


Use tar:

tar cvzf - -T list_of_filenames | ssh hostname tar xzf -
  • 2
    Heres the file format: "In c mode, tar will read names to be archived from filename. The special name -C'' on a line by itself will cause the current directory to be changed to the directory specified on the following line. Names are terminated by newlines unless --null is specified. Note that --null also disables the special handling of lines containing -C''." – Kousha Mar 4 '10 at 10:46

I would like to use scp for that, but scp only allows for transferring one file at a time.

I'm pretty sure that isn't true, at least not for the scp command provided by the OpenSSH included with most Linux distributions.

I use scp file1 file2 ... fileN user@host:/destination/directory/ fairly frequently.

For transparent compression, the SSH protocol has this built in and scp can use it if you provide the -C option on the command line. For lots of similar small files you will find the tar+gz option suggested by akira gains better compression as it can make use of similarity between such files where scp compresses each file as a separate entity. I generally prefer to use scp though as it is easier to resume a partial transfer (or, though I know in this situation the questioner doesn't have the option) rsync as that is both even easier to resume and shares the tar+gz option's whole-stream compression advantage.

  • This is the right answer, but it really only transfers one file at a time. Nevertheless it will make full use of available bandwidth and should run as fast as possible. – Benjamin Bannier Apr 28 '11 at 19:01

Doesn't scp -r yourdir otherhost:/otherdir work?

Try this then:

tar cfz - . | ssh otherhost "cd /mydir; tar xvzf -"

the z-flag to tar does compression. Or you can use -C to ssh:

tar cf - . | ssh -C otherhost "cd /mydir; tar xvf -"
  • what does -C do? – Nathan Fellman Mar 4 '10 at 8:53
  • 1
    The problem is, I have a list of files. The files are spread out over a lot of directories, and not all the files in the directories I want to copy – Peter Smit Mar 4 '10 at 9:34
  • 2
    @Nathan: -C is to lett ssh do the compression. – Jimmy Hedman Mar 4 '10 at 11:32

On RHEL5, here is the only way possible that I know of to secure copy a list of files that contain special characters (like a space). Make a shell script and include the following:

while read line; do
scp "$line" username@servername:/destination/
done < $FILE

It can be done w/o much trouble.

I simply create a list of files with the hostname next to each file and scp seems to parse it fine. ie.

myfiles="${srchost}:~/.bashrc ${srchost}:~/.bash_profile ${srchost}:~/.vimrc ${srchost}:~/.viminfo ${srchost}:~/.toprc ${srchost}:~/.dir_colors"
scp -v $myfiles $HOME

Scp seems to work fine with this syntax. As I attempted it without the hostname next to each file it failed after the first file. I seem to recall another syntax I used for this before and will post it once I locate it.

  • This works, but is quite slow - it seems to start a connection for each file... (I came here looking for a solution for the slow copy...) ~/ can mostly be left out - SCP operates relative to the home directory... If you have a list from elsewhere in $list: scp $(echo "$list"|sed "s/^/$srchost:/") local_destination (This will break for filenames with spaces though) – Gert van den Berg Oct 6 '16 at 8:50

I would recommend to parts use a command like this to create an archive from your file list

tar -cvf allfiles.tar -L mylist.txt

Then use scp to download the archive.

scp user@remotehost:~/allfiles.tar


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.