Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Suppose I have a 10MB text file foo.txt, and it has 100,000 lines. Now, I want to process foo.txt window by window, with a window size 10.

My current script is like this:

for ((i=0;i<$lines;i=i+$step))
do    
    head -$((i+step)) $1 | tail -$step > tmp1
    head -$((i+step)) $2 | tail -$step > tmp2
    setstr=$setstr' '`./accuracy.sh tmp1 tmp2`
done
echo $setstr | awk '{for (i=1;i<=NF;i++) sum+=$i; }END{print sum/NF}'

But it runs slowly. Is there any simple and more efficient way to do this?

share|improve this question

migrated from stackoverflow.com Nov 11 '12 at 12:08

This question came from our site for professional and enthusiast programmers.

add comment

4 Answers

You can do that with split:

Here's an example of how to use it:

split -l 10 input_file output_file_prefix_

The -l option stands for --lines=

And this will split input_file into chunks with 10 lines each, into these files:

output_file_prefix_aa
output_file_prefix_ab
output_file_prefix_ac
...

and so on.

For other ways you can use split, see man split or here

share|improve this answer
    
thanks, sampson. One problem with your solution is that it will produce large amount of temporal files. In my example, it would be 10,000 files. Last time, I used Java to split the file, and it took me long time to delete them. –  JackWM Nov 10 '12 at 21:22
    
@JackWM How did you go about deleting the temporary files? Was it something like rm output_file_prefix_*? –  sampson-chen Nov 10 '12 at 21:26
    
rm would report errors due to the large amount of files. And even though we circumvent this, the deleting still takes much time. –  JackWM Nov 10 '12 at 21:30
add comment

It would be useful to have a bit more context as to your ultimate goal rather than a code snippet. In particular, do you have any control over accuracy.sh?

Anyway, if you want to keep using bash, then you could do

for ((i=0;i<$lines;i+=$step))
do
  let end=i+10
  sed -n $i,${end}p $1 >tmp1
  sed -n $i,${end}p $2 >tmp2
  ...
done
share|improve this answer
add comment

Not sure why this got migrated from StackOverflow. While split is a superuser-style answer, the question was about programming. For example, here's an answer that implements what you're looking for in awk.

One of the really handy aspects of awk is the way it handles pipes.

#!/usr/bin/awk -f

BEGIN {
  cmd="/path/to/handler"
}

{
  print | cmd
}

NR % 10 == 0 {
  close(cmd)
}

Your cmd will be re-opened if it's closed ... and gets closed every 10th line, to be reopened the next line of output.

The effect will be to run handler every 10 lines of input. At the end of the file, handler will be run with whatever lines are left, as cmd is automatically closed as awk exits.

Strictly speaking, you don't need to use a variable like cmd to store the command ... but it does make it simpler to adjust the command, since you'd otherwise need to watch VERY carefully for typos in your close().

share|improve this answer
add comment

This solution does not use any temporary files. What is does is store every line in a buffer array that can hold ten lines. Every time the line number is divisible by ten, it prints all the lines in the buffer.

The obvious pitfall is when the input file (# lines) is not divisible by ten. The solution is to make checks in an END{} clause. Something like:

$ echo {1..33} | tr \  \\n |\
    awk '{lines=NR} END{ if (lines%10!=0) { print "leftover lines"} }'
leftover lines

# STEP1 use modulo to do something every tenth
$ echo {1..200} |tr \  \\n |\
    awk '{a[NR%10]=$0; if (NR%10==0) {print "ten"} }' | cat -n
     1  ten
     2  ten
     3  ten
     4  ten
     5  ten
     6  ten
     7  ten
     8  ten
     9  ten
    10  ten
    11  ten
    12  ten
    13  ten
    14  ten
    15  ten
    16  ten
    17  ten
    18  ten
    19  ten
    20  ten

# STEP 2 do something with every line
$ echo {1..10} | tr \  \\n | awk '{ b+=$0} END {print b}'
55

# putting it together
$ cat every10.awk
{
        a[NR%10]=$0;
        if (NR%10==0) {
                for (i in a) {
                        printf "%s+", a[i]
                        b+=a[i];
                }
                print "0=" b;
                b=0
        }
}
$ echo {1..200} | tr \  \\n | awk -f every10.awk  | column -s= -t
4+5+6+7+8+9+10+1+2+3+0                     55
14+15+16+17+18+19+20+11+12+13+0            155
24+25+26+27+28+29+30+21+22+23+0            255
34+35+36+37+38+39+40+31+32+33+0            355
44+45+46+47+48+49+50+41+42+43+0            455
54+55+56+57+58+59+60+51+52+53+0            555
64+65+66+67+68+69+70+61+62+63+0            655
74+75+76+77+78+79+80+71+72+73+0            755
84+85+86+87+88+89+90+81+82+83+0            855
94+95+96+97+98+99+100+91+92+93+0           955
104+105+106+107+108+109+110+101+102+103+0  1055
114+115+116+117+118+119+120+111+112+113+0  1155
124+125+126+127+128+129+130+121+122+123+0  1255
134+135+136+137+138+139+140+131+132+133+0  1355
144+145+146+147+148+149+150+141+142+143+0  1455
154+155+156+157+158+159+160+151+152+153+0  1555
164+165+166+167+168+169+170+161+162+163+0  1655
174+175+176+177+178+179+180+171+172+173+0  1755
184+185+186+187+188+189+190+181+182+183+0  1855
194+195+196+197+198+199+200+191+192+193+0  1955

The idea here is to use awk print blocks of ten lines and process that, or process with awk directly if the operation is simple arithmetic or string operations.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.