Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Suppose I have a 10MB text file foo.txt, and it has 100,000 lines. Now, I want to process foo.txt window by window, with a window size 10.

My current script is like this:

for ((i=0;i<$lines;i=i+$step))
    head -$((i+step)) $1 | tail -$step > tmp1
    head -$((i+step)) $2 | tail -$step > tmp2
    setstr=$setstr' '`./ tmp1 tmp2`
echo $setstr | awk '{for (i=1;i<=NF;i++) sum+=$i; }END{print sum/NF}'

But it runs slowly. Is there any simple and more efficient way to do this?

share|improve this question

migrated from Nov 11 '12 at 12:08

This question came from our site for professional and enthusiast programmers.

You can do that with split:

Here's an example of how to use it:

split -l 10 input_file output_file_prefix_

The -l option stands for --lines=

And this will split input_file into chunks with 10 lines each, into these files:


and so on.

For other ways you can use split, see man split or here

share|improve this answer
thanks, sampson. One problem with your solution is that it will produce large amount of temporal files. In my example, it would be 10,000 files. Last time, I used Java to split the file, and it took me long time to delete them. – JackWM Nov 10 '12 at 21:22
@JackWM How did you go about deleting the temporary files? Was it something like rm output_file_prefix_*? – sampson-chen Nov 10 '12 at 21:26
rm would report errors due to the large amount of files. And even though we circumvent this, the deleting still takes much time. – JackWM Nov 10 '12 at 21:30

It would be useful to have a bit more context as to your ultimate goal rather than a code snippet. In particular, do you have any control over

Anyway, if you want to keep using bash, then you could do

for ((i=0;i<$lines;i+=$step))
  let end=i+10
  sed -n $i,${end}p $1 >tmp1
  sed -n $i,${end}p $2 >tmp2
share|improve this answer

Not sure why this got migrated from StackOverflow. While split is a superuser-style answer, the question was about programming. For example, here's an answer that implements what you're looking for in awk.

One of the really handy aspects of awk is the way it handles pipes.

#!/usr/bin/awk -f


  print | cmd

NR % 10 == 0 {

Your cmd will be re-opened if it's closed ... and gets closed every 10th line, to be reopened the next line of output.

The effect will be to run handler every 10 lines of input. At the end of the file, handler will be run with whatever lines are left, as cmd is automatically closed as awk exits.

Strictly speaking, you don't need to use a variable like cmd to store the command ... but it does make it simpler to adjust the command, since you'd otherwise need to watch VERY carefully for typos in your close().

share|improve this answer

This solution does not use any temporary files. What is does is store every line in a buffer array that can hold ten lines. Every time the line number is divisible by ten, it prints all the lines in the buffer.

The obvious pitfall is when the input file (# lines) is not divisible by ten. The solution is to make checks in an END{} clause. Something like:

$ echo {1..33} | tr \  \\n |\
    awk '{lines=NR} END{ if (lines%10!=0) { print "leftover lines"} }'
leftover lines

# STEP1 use modulo to do something every tenth
$ echo {1..200} |tr \  \\n |\
    awk '{a[NR%10]=$0; if (NR%10==0) {print "ten"} }' | cat -n
     1  ten
     2  ten
     3  ten
     4  ten
     5  ten
     6  ten
     7  ten
     8  ten
     9  ten
    10  ten
    11  ten
    12  ten
    13  ten
    14  ten
    15  ten
    16  ten
    17  ten
    18  ten
    19  ten
    20  ten

# STEP 2 do something with every line
$ echo {1..10} | tr \  \\n | awk '{ b+=$0} END {print b}'

# putting it together
$ cat every10.awk
        if (NR%10==0) {
                for (i in a) {
                        printf "%s+", a[i]
                print "0=" b;
$ echo {1..200} | tr \  \\n | awk -f every10.awk  | column -s= -t
4+5+6+7+8+9+10+1+2+3+0                     55
14+15+16+17+18+19+20+11+12+13+0            155
24+25+26+27+28+29+30+21+22+23+0            255
34+35+36+37+38+39+40+31+32+33+0            355
44+45+46+47+48+49+50+41+42+43+0            455
54+55+56+57+58+59+60+51+52+53+0            555
64+65+66+67+68+69+70+61+62+63+0            655
74+75+76+77+78+79+80+71+72+73+0            755
84+85+86+87+88+89+90+81+82+83+0            855
94+95+96+97+98+99+100+91+92+93+0           955
104+105+106+107+108+109+110+101+102+103+0  1055
114+115+116+117+118+119+120+111+112+113+0  1155
124+125+126+127+128+129+130+121+122+123+0  1255
134+135+136+137+138+139+140+131+132+133+0  1355
144+145+146+147+148+149+150+141+142+143+0  1455
154+155+156+157+158+159+160+151+152+153+0  1555
164+165+166+167+168+169+170+161+162+163+0  1655
174+175+176+177+178+179+180+171+172+173+0  1755
184+185+186+187+188+189+190+181+182+183+0  1855
194+195+196+197+198+199+200+191+192+193+0  1955

The idea here is to use awk print blocks of ten lines and process that, or process with awk directly if the operation is simple arithmetic or string operations.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .