14

I have 1.6 million files on server A, and only about 20k of them need to get to server B. The destination, server B, is on GoDaddy shared hosting, so I'm just about limited to scp for transferring many files at once.

I'd like to generate a .txt file of those 20k+ files from an SQL query, then feed that list into scp. Are there any options to do so?

cat /proc/version gives me Linux version 2.6.32-531.23.3.lve1.2.65.el6.x86_64 (mockbuild@koji.cloudlinux.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Tue Aug 19 10:37:27 EDT 2014

3
  • 1
    So the file names are stored in a sql database? Also is rsync available? This would be preferred. If the scp dies in the middle of the transfer you'll have to figure out what transferred and what didn't. rsync will do that part for you. Dec 11, 2014 at 16:54
  • @NathanPowell rsync was my first thought too, but GoDaddy's got a whole lot of lockdowns on installing new software through their SSH. Yeah the file paths are currently in an SQL database but I can extract those to a text file easily. Dec 11, 2014 at 17:38
  • Another simple way is to create links to those files in a new directory. Then use rsync -L to copy the files to the remote server. The main downside is that you need to clean up the directory afterwards. I don't think this is a formal answer to this question, so that is why I added as comment.
    – lepe
    May 7, 2019 at 2:54

5 Answers 5

17

if you have the text file already created you do the following

cat /location/file.txt | xargs -i scp {} user@server:/location

this will go line by line of the output of the file.txt and run the scp comamand per line; I hope this helps.

2
  • 1
    That's getting really close, thank you! I'd like to have this process hands-off though, and I'm afraid if it runs line-by-line that it will ask for my password on the remote server every line. I tried SSH keys but to no success, maybe I just wasn't doing it right. Would SSH keys be my only option for that process to be hands off? Dec 11, 2014 at 17:40
  • 1
    This worked wonderfully! What I did instead of xargs on the scp call was use xargs on a cp on the remote server to isolate the files to a directory, somewhat like: cat files.txt | xargs -l bash -c 'cp $0 $1', then used a single scp command to pull that directory. Thank you! Dec 12, 2014 at 8:39
12

Consider rsync as an alternative, which has ssh support built-in and supports reading from a list of files using --files-from.

Example:

rsync -av --progress --partial-dir=/tmp --files-from=file_list.txt . me@example.com:/dest

Advantages over scp:

See also:

2

I wanted to do the same for list of folders and none of the answers using xargs worked for me. In the end I solved this simply with:

scp -r user@server:/location/{"$(cat folder_list.txt | tr '\n' ',' | sed 's/,$//')"} .

Where I pass my list of files to shell expansion. I think this may be helpful to anyone looking for alternative solution.

1

The answer provided works great. I used xargs -a to send the file to standard out. I then combined that with the original answer of cat /location/file.txt | xargs -i scp {} user@server:/location

My Answer:

xargs -a /location/file.txt | xargs -i scp {} user@server:/location
1
  • 1
    What is the benefit of xargs -a over using cat? cat is more portable as the availability of the -a for xargs is system dependent.
    – ephsmith
    Aug 17, 2016 at 15:17
1

The above answers require you to authenticate for every file transferred. SCP runs on top of SSH and GoDaddy provides access to other commands over SSH such as tar. As such, the best method is to use tar with ssh via:

tar -T <file> -cf - | ssh <user>@<host> "tar -C <dir> -xvf -"

If the list of files is on STDIN you can do the following instead: Note: cat is used here for generating STDIN only

cat <file> | tar -T - -cf - | ssh <user>@<host> "tar -C <dir> -xvf -"

For reduced transfer time at the cost of additional processing time, you can add -z or -j to both tar commands to enable gzip or bzip compression respectively.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.