1

Consider following example:

echo abc=123 | sed "s/\([^=]*\)=\(.*\)/\1=\2/"

outputs

abc=123

ie. the same as input. We matched everything up until symbol = as capturing group 1, and everythying after symbol = as capturing group 2. This is just to explain the example.

I can invoke subcommand from sed like this:

echo abc=123 | sed "s/\([^=]*\)=\(.*\)/\1=$(date)/"

producing:

abc=Tue Nov 17 08:35:13 PM CET 2020

so I can call zero-arity function to get replacement.

Question: What if I'd like to take capturing group content and call some command on it to get actual replacement? Say:

echo abc=123 | sed "s/\([^=]*\)=\(.*\)/\1=$('\2 * 2' | bc)/"

producing:

bash: \2 * 2: command not found
abc=123

What I need to capture some text, and run some command on it to get replacement. How to do that?

2
  • you are correct, when creating minimal example I dropped .* in pattern, will update. Thanks. Nov 18, 2020 at 7:42
  • I tried to read into example of GNU sed, which I could work with, and saw some oversimplistic exapmles, but I have no idea how to make it work even in this simple example. If you know how to write that, I'd be grateful to learn that. Nov 18, 2020 at 7:45

1 Answer 1

0

$(date) does not "invoke subcommand from sed". The shell expands double-quoted $(…) before sed is run. The output from date appears in the command and this is what sed gets. Your command

echo abc=123 | sed "s/\([^=]*\)=\(.*\)/\1=$(date)/"

became this:

echo abc=123 | sed "s/\([^=]*\)=\(.*\)/\1=Tue Nov 17 08:35:13 PM CET 2020/"

and only then sed started. With this approach you cannot pass anything from sed itself to date (or bc, or whatever) because the inside of $(…) finishes before sed starts.

Standard sed cannot do what you want. GNU sed can thanks to the e flag. Let's build an example with GNU sed and bc.

echo abc=123 | sed "s/\([^=]*\)=\(.*\)/printf '%s' '\1='; printf '%s\n' '\2 * 2' | bc/e"
#                                      ----------------------------------------------

Without e the underlined fragment would be a replacement. With e it gets executed in a shell and its output becomes the replacement.

\n is there because in my tests bc didn't work with incomplete lines. Note this \n is interpreted by sed itself, printf will get an actual newline character. A point when printf gets \n to interpret is with %s\\\\n. This is because your current shell changes \\\\n in a double-quoted string to \\n, sed interprets \\ as \, the inner shell gets single-quoted \n and only then printf gets \n (more levels than in this question, but similar). Having \n interpreted by sed does not break the command so you can stick to this simple form.

It would be nice if you could leave the abc= part intact and only replace 123. For this a lookbehind feature is useful, but sed does not support it. One of the answers to the linked question advises a capture group and a backreference in the replacement string, this is exactly what we did with \1.


One big problem with running a shell command this way is there's a code injection vulnerability. In the shell-command-to-be we single-quoted \1 and \2. If sed substitutes any of them with a string containing ' the inner shell will close the quoting prematurely. Try the following command (the sed part is the same as above):

echo "abc=123'; rm -i /very/important/file'" \
 | sed "s/\([^=]*\)=\(.*\)/printf '%s' '\1='; printf '%s\n' '\2 * 2' | bc/e"

(I used rm -i in case you really have /very/important/file in your system.)

You need to be sure the input won't inject code. Or you need to rewrite the regex ([^=]* and .* parts) so possibly dangerous ' cannot get from arbitrary input to the shell command. Note if we single-quoted the entire sed expression and double-quoted \1 and \2, the double-quotes would get to the inner shell and then " (not ') in the input would be dangerous, along with $var, $(code) and such! And if we left \1 or \2 unquoted in the context of the inner shell then it would be even worse.

In general one should pass arbitrary data to a shell as command line arguments, not in a command string. E.g. with find -exec one can do this right. Code injection is then impossible, unless the actual (static) shell command executes a positional parameter or mishandles something badly. Unfortunately while invoking a shell from GNU sed all you can pass is a command string.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .