I read that xargs was good for processing the output of a command one line at a time (and it is). I have the following line in my script.

./gen-data | awk '{printf $2 " "; printf $1=$2=$3=""; gsub (" ", "", $0);if(length($0) == 0){ print "0000"} else{print $0}}' | xargs -t -n2 -P1 bash -c 'datatojson "$@"' _

It produces the right output, there is no question of that. However, gen-data produces something like 1000 lines, and what I really would like is for this command to execute after each line, not after 1000 lines (It's clearly stopping regularly to get more input).

Here is what gen-data looks like:

candump $interface &
while true; do
    while read p; do
        cansend $interface $(echo $p | awk 'NF>1{print $NF}');
    done < <(shuf $indoc)

(cansend sends data to an interface and candump reads from that interface and outputs it onto the screen, but I wager that's not too relevant). In any case candump seems to be continuously streaming output, but when I pipe that into awk and xargs, it becomes chunked. Is it just because I used shuf? I would think that since it's going through the interface, and being read on the other side, it would be less chunked than shuf provides.

  • Wy are you using shuf? Does this script work without using it? (I know it should, but it is necessary to rule it out) – Felipe Lema Jul 27 '15 at 14:33
  • @FelipeLema I just tried it with cat instead of shuf and it's doing the same thing. Not sure how else I can feed it into that loop, I'm not very experienced in scripting. – user3475234 Jul 27 '15 at 15:09

You can try the same command, this time using multiple hacks to avoid buffering:

./gen-data | gawk '{printf $2 " "; printf $1=$2=$3=""; gsub (" ", "", $0);if(length($0) == 0){ print "0000"} else{print $0}; fflush(stdout)}' | stdbuf -o0 xargs -t -n2 -P1 bash -c 'datatojson "$@"' _

Mind the change from awk to gawk and the use of fflush. You can also try mawk -Winteractive. Also mind that I added stdbuf -o0 before xargs. You can also try the latest at the beginning with ./gen-data

  • Worked perfectly. What does the -o0 do on the stdbuf command? – user3475234 Jul 27 '15 at 15:28
  • 1
    As stated in the stdbuf man page: sets the buffering of the command as "unbuffered" – Felipe Lema Jul 27 '15 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.