Using scp to transfer a .txt file list of files

Question

I have 1.6 million files on server A, and only about 20k of them need to get to server B. The destination, server B, is on GoDaddy shared hosting, so I'm just about limited to scp for transferring many files at once.

I'd like to generate a .txt file of those 20k+ files from an SQL query, then feed that list into scp. Are there any options to do so?

cat /proc/version gives me Linux version 2.6.32-531.23.3.lve1.2.65.el6.x86_64 (mockbuild@koji.cloudlinux.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Tue Aug 19 10:37:27 EDT 2014

Answer 1

if you have the text file already created you do the following

cat /location/file.txt | xargs -i scp {} user@server:/location

this will go line by line of the output of the file.txt and run the scp comamand per line; I hope this helps.

Answer 2

Consider rsync as an alternative, which has ssh support built-in and supports reading from a list of files using --files-from.

Example:

rsync -av --progress --partial-dir=/tmp --files-from=file_list.txt . me@example.com:/dest

Advantages over scp:

• Avoid cat/xargs and command line length limit, so only enter password once
• Resume progress after being interrupted
• Delta transfer algorithm avoids copying files that already exist
• Generally more robust and featureful
• Further reading: How does scp differ from rsync?

See also:

• Do you need -e ssh for rsync?

Answer 3

I wanted to do the same for list of folders and none of the answers using xargs worked for me. In the end I solved this simply with:

scp -r user@server:/location/{"$(cat folder_list.txt | tr '\n' ',' | sed 's/,$//')"} .

Where I pass my list of files to shell expansion. I think this may be helpful to anyone looking for alternative solution.

Answer 4

The answer provided works great. I used xargs -a to send the file to standard out. I then combined that with the original answer of cat /location/file.txt | xargs -i scp {} user@server:/location

My Answer:

xargs -a /location/file.txt | xargs -i scp {} user@server:/location

Answer 5

The above answers require you to authenticate for every file transferred. SCP runs on top of SSH and GoDaddy provides access to other commands over SSH such as tar. As such, the best method is to use tar with ssh via:

tar -T <file> -cf - | ssh <user>@<host> "tar -C <dir> -xvf -"

If the list of files is on STDIN you can do the following instead: Note: cat is used here for generating STDIN only

cat <file> | tar -T - -cf - | ssh <user>@<host> "tar -C <dir> -xvf -"

For reduced transfer time at the cost of additional processing time, you can add -z or -j to both tar commands to enable gzip or bzip compression respectively.