Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a huge data source that I'm filtering using some greps.

Here's basically what I'm doing right now:

#!/bin/bash
param1='something'
param2='another'
param3='yep'
echo $(avro-read /log/huge_data | grep $param1 | grep "$param2-" | grep $param3 | wc -l) / $(avro-read /log/ap/huge_data | grep $param1 | grep -v "$param2-" | grep $param3 | wc -l) | bc -l

Notice how I'm doing mostly the same filtering twice (a single difference the second time), taking the count of each, and dividing the final result. This is definitely a hacky thing to do, but I'd like to try and speed it up just a bit and only perform the initial filtering once without using a temp file.

I tried using a fifo, but I'm not sure if it's possible to have two processes in one script reading from it, as well as have a third process "wait" until both are done to compute the final result. I also looked into using tee, but again not sure how to synchronize the resulting sub processes.

EDIT: Solved this myself using http://superuser.com/a/561248/43649, but marked another suggestion as the answer.

share|improve this question
    
No; once one process reads data from the fifo, that data is gone and will not be seen by the other process. – chepner Mar 5 '13 at 17:04
up vote 2 down vote accepted

If you just want to avoid creating temporary files (or storing the output of grep in a variable), you can feed it to a for loop like this:

#!/bin/bash

IFS=$'\n'
yay=0
nay=0

for line in `avro-read /log/huge_data | grep $param1 | grep $param3`; do
    [[ $line =~ $param2- ]] && yay=$(($yay + 1)) || nay=$(($nay + 1))
done

echo $yay / $nay \* 100 | bc -l

unset IFS

I've created a modified version of the approach in your self-answer that won't require temporary files:

#!/bin/bash

(avro-read /log/huge_data | grep $param1 | grep $param3 | tee \
     >(echo yay=`grep -c "$param2-"`) \
     >(echo nay=`grep -vc "$param2-"`) \
     >/dev/null | cat ; echo 'echo $yay / $nay \* 100 | bc -l') | sh

The output of the individual grep -c commands and the echo command get printed as

yay=123
nay=456
echo $yay / $nay \* 100 | bc -l

to avoid race conditions1. Piping to sh executes the printed commands.

1 Whichever grep -c command finishes first will print the first line of output.

share|improve this answer
    
Super cool way to loop through the data. – Andrew Mar 5 '13 at 18:25
    
Great -- that's a "need to remember" trick to get the values out of the process substitutions :) – mpy Mar 5 '13 at 20:37
    
Wow! Super cool way of doing it. Thanks for adapting my answer! There is definitely a lot to learn from this. – Andrew Mar 11 '13 at 22:25

I ended up solving this like so:

#!/bin/bash
param1='something'
param2='another'
param3='yep'

avro-read /log/huge_data | grep $param1 | grep $param3 \
| tee \
>(grep "$param2-" | wc -l | tr -d '\n' > has_count) \
>(grep -v "$param2-" | wc -l | tr -d '\n' > not_count) \
> /dev/null

echo $(cat has_count | tr -d '\n') '/' $(cat not_count | tr -d 'n') '* 100' | bc -l

So rather than relying on a fifo or temp file, I used tee to split the stream into two separate processes that just output a count! This way I don't need to try and synchronize the two processes before trying to divide the counts.

share|improve this answer
    
You are using two temp files... – mpy Mar 5 '13 at 18:28
    
Very interesting! I've included a modification of your approach in my answer that doesn't require temporary files. – Dennis Mar 5 '13 at 19:24
    
By "not relying on a temp file" in this case, I meant a temp file full of actual data. In this case, it's just storing a number. – Andrew Mar 11 '13 at 22:22

Hm, zsh has a feature, called MULTIOS. Therewith it's possible to connect one process to two fifo's. If that's an option here a small demo:

#!/bin/zsh -f

setopt multios

mkfifo f1 f2 2> /dev/null

param1='something'
param2='another'
param3='yep'

{ avro-read /log/huge_data | grep $param1 | grep $param3 } > f1 > f2 &

( cat f1 | grep $param2 | wc -l > value1 ) &!
value2=$(cat f2 | grep -v $param2 | wc -l)

print $(( 1. * $( cat value1 ) / $value2 ))

rm value1

However, I could not figure out a way to get round the creation of the temporary file value1, which should be probably avoided as pointed out by Dennis. But perhaps you'll like this solution nevertheless.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .