Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a file where each line is a file name. Content of the file:

file1
file2
file3
...

I then run a grep command to get a subset of the list of the file names.

How do I feed the output of the grep command into scp? I have played around with xargs and $() without much success...

share|improve this question

2 Answers 2

up vote 1 down vote accepted

No need to use xargs:

for file in $(grep <some-pattern> <filelist>); do scp $file <remote>; done;

Where

  • <some-pattern> is your grep pattern
  • <filelist> is your list of files
  • <remote> is your remote end of the scp command

The for loop will iterate over each line produced by grep and pass this to $file, which you can re-use in the commands executed between do and done.

Also see: Advanced Bash-Scripting Guide: – Loops

share|improve this answer
    
perfect, thanks! –  dognar Jul 20 '11 at 11:46
1  
Dont forget to quote your filenames (in case of spaces). –  SamK Jul 20 '11 at 12:31
1  
If any of the file names contains whitespace or globbing characters, this won't work. –  Gilles Jul 20 '11 at 13:52

To iterate over a list of file names with one name per line, you have several possibilities:

  • Use command substitution to work the file into a list of file names inside the shell. You need to take precautions, because you can't put the command substitution in double quotes, since the result of the command substitution needs to be split into words. You need to turn off globbing and set IFS to only split words on newlines.

    set -f; IFS='
    '
    scp -- $(…) remote.example.com:
    
  • Use the read built-in to process the file names in a loop. Again, you need to be careful, because read treats backslashes and $IFS characters specially.

    … | while IFS= read -r x; do scp -- "$x" remote.example.com: done
    
  • You can use GNU xargs, but you have to tread carefully, because xargs expects its input in an oddly quoted format. You must pass the -d '\n' option to explicitly select newlines as the input separator and turn off the special behavior of whitespace and \'".

    … | xargs -d '\n' -I {} scp {} remote.example.com:
    

    The only advantage of xargs is that it can group multiple arguments onto one invocation up to the command line length limit, but that only works if the arguments are passed last to the command. You can achieve that through an intermediate call to sh, but it's rarely worth the trouble.

    … | xargs -d '\n' sh -c 'scp -- "$@" "$0"' remote.example.com:
    
  • An easier way to use xargs is to get a null-separated list instead of a newline-separated. This requires the -0 option, which is supported by GNU xargs and also recent *BSD. As above, shoving the arguments into a non-final position requires a bit of work.

    … | tr '\n' '\0' | xargs -0 sh -c 'scp -- "$@" "$0"' remote.example.com:
    

The part would be grep NAME_REGEXP /path/to/filenames.list if you meant to filter the file names with a regular expression. If you meant to filter on the file contents, use one of the techniques above to feed the file names to grep, e.g.

set -f; IFS='
'
scp -- $(grep -l CONTENT_REGEXP -- $(cat /path/to/filenames.list)) \
    remote.example.com:

Another possibility that bypasses the shell difficulties is to use rsync and its --files-from option. Assuming the file names are all relative to the current directory:

grep NAME_REGEXP /path/to/filenames.list |
rsync -a --files-from - . remote.example.com:
share|improve this answer
    
Very elaborate, I'll give you that! (Just wondering, does that still justify downvoting if it worked for the OP?) –  slhck Jul 20 '11 at 13:55
    
@slhck Any answer that's wrong (and I don't just mean wrong to detail, if it had been just a matter of adding a quote or two I'd just have edited your answer) is a legitimate downvoting target, in fact it should be downvoted. An answer that looks right when you test it on a simple example but breaks in real-world scenarios is the worst kind. –  Gilles Jul 20 '11 at 14:01
    
Well, imo it's not "wrong" when it does work -- even if it might be a simple scenario. The button actually says, "This answer is not useful", and it obviously was useful. I knew it wasn't gonna work with complex file names, and by adding your comment above, that was absolutely clarified. But yeah, let's leave it at that, not going to fight over it! –  slhck Jul 20 '11 at 14:08
    
Thanks for your detailed answer Gilles, I'm sure it will be handy if I come across complex filenames. –  dognar Jul 20 '11 at 21:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.