TL;DR: Need to reuse stream in sed so that extracted string is added into current line's ending.

I have have a script to extract a substring using sed and make a command to copy using this substring's name.

There are few things I tried such as xargs, but sed -e parses each line separately and reinvoking sed reads all lines again for every line currently being parsed:

This line below just prints out the string extracted:

cat hello.txt | sed -e 's/.*search_start\(.*\)search_end*/\1/' | xargs -I@ "echo ./@"

This line creates a string with copy command added for folder location:

cat hello.txt | sed -e 's/\(^\)/copy /; s/$/ .\/location/;'

However, I want to add these together so that the extracted_str searched string can be referenced in place of location folder here. So, I need something like this (for each line):

copy input_sed_line extracted_str
  • Please edit your question to show an example of what hello.txt contains and what you want to happen. – Scott Apr 6 at 2:05

If I understand the question right, you need an extra occurrence of the whole line along with whatever sed extracts from it. Well, sed can print the whole line with p, so this is possible:

< hello.txt sed -e 'p; s/.*search_start\(.*\)search_end.*/\1/' | xargs -d '\n' -n 2 copy --


  • Whatever copy is (did you mean cp?).
  • I assumed copy understands -- (the end of options marker).
  • I fixed the useless use of cat.
  • I assumed search_end* should be search_end.*.
  • -d and -n options of xargs are not required by POSIX. See man 1 xargs in your OS to tell if you can use them. Here -d '\n' specifies the newline character as delimiter, -n 2 tells xargs to use exactly two input items per copy. -n, if supported, is probably somewhat limited.

The above may work in many cases, but if I were you I would use a shell and its read builtin. It's true one shouldn't loop read to parse text; sed, awk and such are the right tools in general. Here however your goal is not only to parse text. Your goal is to craft and run commands. A shell like sh or bash is the right tool to run commands.


while IFS= read -r wholeline; do
   target="$(printf '%s\n' "$wholeline" | sed -e 's/.*search_start\(.*\)search_end.*/\1/')"
   copy -- "$wholeline" "$target"
done < hello.txt

The code runs a separate sed process for each line. In your case this can be easily avoided:


while IFS= read -r wholeline; do
   target="${wholeline##*search_start}"   # removing prefix
   target="${target%%search_end*}"        # removing postfix
   copy -- "$wholeline" "$target"
done < hello.txt

One could do (almost) the same with awk:

< hello.txt awk '{
   cmdline="copy -- '"'"'"$0"'"' '"'"target"'"'"'"

There are problems though:

  1. Quoting frenzy. There are three levels that need quotes and interpret them: the original shell where awk is invoked, awk itself, the shell invoked by system(). One can get rid of the first level by building an awk script (with the shebang like #!/usr/bin/awk). Still it's cumbersome. I'm not even sure I placed these quotes exactly as I wanted.
  2. cmdline is passed as a string which is then parsed, not as an array of arguments. If the input contains the literal ' character then things will break, some parts of the input may get executed (code injection!).
  3. system(cmdline) invokes sh, additional separate shell for every single line. You can print cmdline instead, then pipe the whole output to a single sh (or save to a file to parse it later). This will reduce the number of processes but the previous problem will stay: the textual output will be parsed sooner or later.

From my experience I can say these problems are not limited to awk. If you want to use any tool to run an external command with arguments based on arbitrary input, pay attention whether it passes a string (to be parsed by sh or anything) or spawns a new process with strictly defined array of arguments. E.g. find -exec does the latter (or at least common implementations of find do).


  • xargs may lack useful options.
  • Text processing tools are better than shells in processing text, but they are usually a lot worse in running external commands.

For this reasons I think a shell loop is at least reasonable here. Still you should read good answers to the already linked question. Note my shell script doesn't follow all the hints you can find there.

  • Thank you very much for the detailed answer Kamil! This worked great. I had produced an alternate solution using xargs @1, @1 using bash to echo piped commands. But this (sed -e p ) is a better solution. – Vishal Anand Apr 20 at 19:33
  • Adding a tiny detail to your query on copy vs cp, I had been using UNIX binaries to execute on Windows cmd, hence the usage of copy in my question. – Vishal Anand Apr 20 at 19:40

Why not:

location=$(cat hello.txt | sed -e 's/.*search_start\(.*\)search_end*/\1/' | xargs -I@ "echo ./@")
cat hello.txt | sed -e 's/\(^\)/copy /; s/$/ .\/'"$location"'/;'
  • The location is supposed to be different for each line. Correct me if I am wrong, this current command would store just the last line's location – Vishal Anand Apr 4 at 20:44
  • The substring extraction will find all occurences. $location is going to be a multiline variable. You may need to put in a -0 in your xargs though if location is one line. – Davey Apr 4 at 22:15
  • @Davey - if it's a "multiline variable", then $location will expand to such in the second line... which is broken? – Attie Apr 5 at 10:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.