Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a huge data source that I'm filtering using some greps.

Here's basically what I'm doing right now:

#!/bin/bash
param1='something'
param2='another'
param3='yep'
echo $(avro-read /log/huge_data | grep $param1 | grep "$param2-" | grep $param3 | wc -l) / $(avro-read /log/ap/huge_data | grep $param1 | grep -v "$param2-" | grep $param3 | wc -l) | bc -l

Notice how I'm doing mostly the same filtering twice (a single difference the second time), taking the count of each, and dividing the final result. This is definitely a hacky thing to do, but I'd like to try and speed it up just a bit and only perform the initial filtering once without using a temp file.

I tried using a fifo, but I'm not sure if it's possible to have two processes in one script reading from it, as well as have a third process "wait" until both are done to compute the final result. I also looked into using tee, but again not sure how to synchronize the resulting sub processes.

EDIT: Solved this myself using http://superuser.com/a/561248/43649, but marked another suggestion as the answer.

share|improve this question
    
No; once one process reads data from the fifo, that data is gone and will not be seen by the other process. –  chepner Mar 5 '13 at 17:04
add comment

3 Answers

up vote 2 down vote accepted

If you just want to avoid creating temporary files (or storing the output of grep in a variable), you can feed it to a for loop like this:

#!/bin/bash

IFS=$'\n'
yay=0
nay=0

for line in `avro-read /log/huge_data | grep $param1 | grep $param3`; do
    [[ $line =~ $param2- ]] && yay=$(($yay + 1)) || nay=$(($nay + 1))
done

echo $yay / $nay \* 100 | bc -l

unset IFS

I've created a modified version of the approach in your self-answer that won't require temporary files:

#!/bin/bash

(avro-read /log/huge_data | grep $param1 | grep $param3 | tee \
     >(echo yay=`grep -c "$param2-"`) \
     >(echo nay=`grep -vc "$param2-"`) \
     >/dev/null | cat ; echo 'echo $yay / $nay \* 100 | bc -l') | sh

The output of the individual grep -c commands and the echo command get printed as

yay=123
nay=456
echo $yay / $nay \* 100 | bc -l

to avoid race conditions1. Piping to sh executes the printed commands.

1 Whichever grep -c command finishes first will print the first line of output.

share|improve this answer
    
Super cool way to loop through the data. –  Andrew Mar 5 '13 at 18:25
    
Great -- that's a "need to remember" trick to get the values out of the process substitutions :) –  mpy Mar 5 '13 at 20:37
    
Wow! Super cool way of doing it. Thanks for adapting my answer! There is definitely a lot to learn from this. –  Andrew Mar 11 '13 at 22:25
add comment

I ended up solving this like so:

#!/bin/bash
param1='something'
param2='another'
param3='yep'

avro-read /log/huge_data | grep $param1 | grep $param3 \
| tee \
>(grep "$param2-" | wc -l | tr -d '\n' > has_count) \
>(grep -v "$param2-" | wc -l | tr -d '\n' > not_count) \
> /dev/null

echo $(cat has_count | tr -d '\n') '/' $(cat not_count | tr -d 'n') '* 100' | bc -l

So rather than relying on a fifo or temp file, I used tee to split the stream into two separate processes that just output a count! This way I don't need to try and synchronize the two processes before trying to divide the counts.

share|improve this answer
    
You are using two temp files... –  mpy Mar 5 '13 at 18:28
    
Very interesting! I've included a modification of your approach in my answer that doesn't require temporary files. –  Dennis Mar 5 '13 at 19:24
    
By "not relying on a temp file" in this case, I meant a temp file full of actual data. In this case, it's just storing a number. –  Andrew Mar 11 '13 at 22:22
add comment

Hm, zsh has a feature, called MULTIOS. Therewith it's possible to connect one process to two fifo's. If that's an option here a small demo:

#!/bin/zsh -f

setopt multios

mkfifo f1 f2 2> /dev/null

param1='something'
param2='another'
param3='yep'

{ avro-read /log/huge_data | grep $param1 | grep $param3 } > f1 > f2 &

( cat f1 | grep $param2 | wc -l > value1 ) &!
value2=$(cat f2 | grep -v $param2 | wc -l)

print $(( 1. * $( cat value1 ) / $value2 ))

rm value1

However, I could not figure out a way to get round the creation of the temporary file value1, which should be probably avoided as pointed out by Dennis. But perhaps you'll like this solution nevertheless.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.