1

I want to split and GZip a large file, and this answer appeared to be what I'm looking for, and it seemed like a very useful way of doing things I never thought of, so I'd like to generalize it; the only problem is: it doesn't appear to work.

Say I want to split my input and process it further (I know split but I want to pipe it around in my script directly!)

This uses read to read a line into a variable

#!/bin/bash
printf " a \n b \n c \n d " |
for ((i = 0 ; i < 2 ; i++)) ; do
  echo "<< $i >>"
  for ((j = 0 ; j < 2 ; j++)) ; do
    read l
    echo "$l"
  done
done

It prints

<< 0 >>
a
b
<< 1 >>
c
d

Which is almost what I want, apart from the fact that it trims the spaces from the start and end (and maybe modifies the line in other ways? Will it work with arbitrary UTF-8 encoded content?) edit solved

And I imagine it might be quite slow. edit Benchmarked it: at least 3000x slower.

So I tried to pipe it through head (I get the result with using awk as the answer suggests, it doesn't appear to do anything differently)

#!/bin/bash
printf " a \n b \n c \n d " |
for ((i = 0 ; i < 2 ; i++)) ; do
  echo "<< $i >>"
  head -n 2
done

That prints

<< 0 >>
 a 
 b 
<< 1 >>

And stops because head apparently closes its input on exit. I haven't found a program that doesn't do this, and maybe it's actually enforced by the system? (I'm on OS X)

Using head -n 2 <&0 which (according to the bash docs) copies the file descriptor first doesn't work either.

Do I have to use a named pipe? Is there some incantation to make this work?

5
  • How do you know that blanks are being stripped? You should put something like echo "..$l.." to see what has been stored in l, as echo ignores leading and trailing blanks.
    – AFH
    Jul 30, 2014 at 16:32
  • read strips it, echo ".$l." prints .a.. I think the shell strips spaces when splitting the arguments, l=" a "; echo $l prints a but l=" a "; echo "$l" prints the spaces ` a ` . (read can also the input by spaces as well and fill multiple variables, that's probably why)
    – pascal
    Jul 30, 2014 at 18:16
  • Are you comfortable using another language, like Perl? It would be easy to read in a file, loop thru the lines, building up output files (or internal data/variables) to hold each of the "chunks". Let me know if you could take an example in Perl and modify it to so you can "pipe" the data around the rest of the script. If you can handle that, I can write the initial Perl to split it up.
    – jimtut
    Jul 30, 2014 at 19:24
  • No, I'd know how to do it in another language (Python for me) but I was hoping there was a simple way to do this in bash that I'm missing. Some "prevent program from closing stdin" flag?
    – pascal
    Jul 30, 2014 at 20:40
  • Sorry, some visitors arrived, and I tried to wrap up my comment, but rather too hastily, it seems. I have found an alternative in the line command: l="`line`" reads one full line from standard input and assigns it to l, complete with all blanks. Each invocation of line will read another input line. You can use this in place of your read l.
    – AFH
    Jul 30, 2014 at 21:23

3 Answers 3

1

The problem here is not exactly that head or awk are "closing the input". They have no choice; any program closes its input when it terminates, and that's enforced by the operating system.

The issue is that the standard input is a pipe, and the programs are doing buffered reads. There's no way to unread from a pipe, so whatever data is in the readahead is gone. If instead of using a pipe you use a file, you'll probably see that it works fine:

#!/bin/bash
printf " %s \n" a b c d > /tmp/abcd
for ((i = 0 ; i < 2 ; i++)) ; do
    echo "<< $i >>"
    for ((j = 0 ; j < 2 ; j++)) ; do
        read
        echo "$REPLY"
    done
done < /tmp/abcd

At least, that works fine on Ubuntu. You can make it work with a pipe if you turn buffering off -- but that is likely to make things really slow. Here's a little C program which turns buffering off and then echoes its input character by character until it consumes the requested number of lines:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
  int n = 1000;
  if (argc > 1) n = atoi(argv[1]);
  setvbuf(stdin, NULL, _IONBF, 0);
  for (int ch = getchar(); ch != EOF; ch = getchar()) {
    putchar(ch);
    if (ch == '\n' && --n <= 0) break;
  }
  return n > 0;
}

That worked fine for me (on Ubuntu, again -- and you need to compile it with -std=c99 or -std=c11 so that the compiler doesn't complain). It's true that the program doesn't call fclose(stdin), but adding won't make any difference. On the other hand, removing the call to setvbuf will probably get you back to the symptom you observed with head. (And it will also make the program run a lot faster.)

If you had GNU split instead of the BSD version which ships with OS X, you'd be able to use the useful --filter=COMMAND syntax which does pretty well exactly what you want; instead of creating split files, it pipes each file section into an invocation of the specified COMMAND (and sets the environment variable $FILE to the expected filename).

1
  • Agh, yes, didn't think about buffering. Didn't know about --filter; that's the kind of flexibility I was looking for.
    – pascal
    Jul 31, 2014 at 14:11
1

By specifying a variable to read you order it to perform a word splitting. Do not do that, and spaces will stay untouched:

#!/bin/bash
printf " a \n b \n c \n d " |
for ((i = 0 ; i < 2 ; i++)) ; do
    echo "<< $i >>"
    for ((j = 0 ; j < 2 ; j++)) ; do
        read
        echo "$REPLY"
    done
done

Output:

<< 0 >>
 a  
 b  
<< 1 >>
 c  
 d  

It seems to be very simple, but actually you’ve asked a very good question since that feature does not explained in the man clearly.

P. S. I would use an -r flag (do not treat \ as escape char) for read also.

4
  • nice, didn't know about $REPLY. The alternative I've seen is IFS= read var. However, using read in a bash loop is incredibly slow (1e3 lines take 6.3sec) compared to head (1e6 lines take 1.8sec), so it's only useful for small files.
    – pascal
    Jul 30, 2014 at 22:34
  • @pascal Yes, Bash is not a proper tool to operate large files. I’ve added an answer with another solution. Jul 30, 2014 at 22:57
  • @pascal And no! I’ve missed the point at first look: 1000 lines takes 6 s (not ms?) to be printed, you said? You are doing something extremely wrong. Jul 30, 2014 at 23:21
  • Some of the lines are quite long I think…
    – pascal
    Jul 31, 2014 at 1:08
0

But if you want to write a stand-alone script to operate large files, AWK would be much more suitable than Bash by reasons of efficiency. An one-liner:

$ awk 'NR%2 { print "<< " int(NR/2) " >>" }; 1' <<< $' a \n b \n c \n d '
<< 0 >>
 a 
 b 
<< 1 >>
 c 
 d 

The same as a script:

#!/usr/bin/awk -f

# where (number of line) mod 2 == 1, i. e. every odd line
NR%2 == 1 {
    # print (number of line) div 2
    print "<< " int(NR/2) " >>"
}

{  
    # print input stream
    print
} 

The very same as a Bash-script:

#!/bin/bash

while read; do
    let lnum++
    ((lnum % 2 == 1)) && \
        echo "<< $((lnum / 2)) >>"
    echo "$REPLY"
done

A benchmark with one million lines:

$ awk 'BEGIN { for (i=1; i<=10^6; i++) print i }' >> 1e6

$ time ./pascal.awk < 1e6 > /dev/null

real    0m0.663s
user    0m0.656s
sys     0m0.004s

$ time ./pascal.sh < 1e6 > /dev/null

real    0m31.293s
user    0m29.410s
sys     0m1.852s

You see, why Bash is not a preferable interpreter here.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.