How to feed grep output into scp?

Question

I have a file where each line is a file name. Content of the file:

file1
file2
file3
...

I then run a grep command to get a subset of the list of the file names.

How do I feed the output of the grep command into scp? I have played around with xargs and $() without much success...

Answer 1

No need to use xargs:

for file in $(grep <some-pattern> <filelist>); do scp $file <remote>; done;

Where

• <some-pattern> is your grep pattern
• <filelist> is your list of files
• <remote> is your remote end of the scp command

The for loop will iterate over each line produced by grep and pass this to $file, which you can re-use in the commands executed between do and done.

Also see: Advanced Bash-Scripting Guide: – Loops

Answer 2

To iterate over a list of file names with one name per line, you have several possibilities:

• Use command substitution to work the file into a list of file names inside the shell. You need to take precautions, because you can't put the command substitution in double quotes, since the result of the command substitution needs to be split into words. You need to turn off globbing and set IFS to only split words on newlines.

  set -f; IFS='
  '
  scp -- $(…) remote.example.com:
  

• Use the read built-in to process the file names in a loop. Again, you need to be careful, because read treats backslashes and $IFS characters specially.

  … | while IFS= read -r x; do scp -- "$x" remote.example.com: done
  

• You can use GNU xargs, but you have to tread carefully, because xargs expects its input in an oddly quoted format. You must pass the -d '\n' option to explicitly select newlines as the input separator and turn off the special behavior of whitespace and \'".

  … | xargs -d '\n' -I {} scp {} remote.example.com:
  

  The only advantage of xargs is that it can group multiple arguments onto one invocation up to the command line length limit, but that only works if the arguments are passed last to the command. You can achieve that through an intermediate call to sh, but it's rarely worth the trouble.

  … | xargs -d '\n' sh -c 'scp -- "$@" "$0"' remote.example.com:
  

• An easier way to use xargs is to get a null-separated list instead of a newline-separated. This requires the -0 option, which is supported by GNU xargs and also recent *BSD. As above, shoving the arguments into a non-final position requires a bit of work.

  … | tr '\n' '\0' | xargs -0 sh -c 'scp -- "$@" "$0"' remote.example.com:
  

The … part would be grep NAME_REGEXP /path/to/filenames.list if you meant to filter the file names with a regular expression. If you meant to filter on the file contents, use one of the techniques above to feed the file names to grep, e.g.

set -f; IFS='
'
scp -- $(grep -l CONTENT_REGEXP -- $(cat /path/to/filenames.list)) \
    remote.example.com:

Another possibility that bypasses the shell difficulties is to use rsync and its --files-from option. Assuming the file names are all relative to the current directory:

grep NAME_REGEXP /path/to/filenames.list |
rsync -a --files-from - . remote.example.com: