Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Consider a stream of one-word lines such as the stdout of

$ echo foo bar baz quux xyxxy thud | tr ' ' '\n'
foo
bar
baz
quux
xyxxy
thud

I say "one-word" to indicate there is no whitespace other than the newlines. I'd like to "columnize" the data by filling in a fixed number of fields (not single characters) per line, one row at a time. For small amounts of data, xargs does this nicely:

$ echo foo bar baz quux xyxxy thud | tr ' ' '\n' | xargs -n 2
foo bar
baz quux
xyxxy thud

However using xargs this way is slow. I have hundreds of gigabytes of text to process, so I would very much like my write throughput to hit the 50 MB/s or so that I know this hardware can do (the xargs above is orders of magnitude slower.)

Is there something better available in the standard toolbox? I can write something if needed but I'd prefer using a text filter that's already there.

share|improve this question
up vote 0 down vote accepted

The best I can come up with is

$ echo foo bar baz quux xyxxy thud | tr ' ' '\n' \
  | perl -lne '$x.=" $_"; if(!($. % 2)){print $x; $x="";} END{print $x if $x}'
 foo bar
 baz quux
 xyxxy thud

or the arguably more elegant, though no shorter

$ echo foo bar baz quux xyxxy thud bang| tr ' ' '\n' \
  | perl -ne 'chomp; print "$_ "; print "\n" if (!($. % 3)); END {print "\n"}'
foo bar baz
quux xyxxy thud
bang

Efficiency?

$ time perl -e 'print "foo\n" for (1..10000)' | xargs -n 3 > /dev/null

real    0m1.330s
user    0m0.500s
sys     0m0.830s


$ time perl -e 'print "foo\n" for (1..10000)' \
  | perl -ne 'chomp; print "$_ "; print "\n" if (!($. % 3)); END {print "\n"}' \ 
  > /dev/null

real    0m0.060s
user    0m0.030s
sys     0m0.030s

Initially I tried 1000000 words but got fed up waiting for xargs though perl was 1.45s

(Timings done on andLinux under Vista-32 on an AMD-64 x2 5600+ 2.8GHz with 4GB RAM)

share|improve this answer
    
The perl one-liner hits about 20 MB/s write to disk (on my hardware), but there's enough CPU left that I can pipeline them. Thanks! – phs Sep 3 '11 at 19:25

pr is vastly underrated.  Here’s what I came up with:

% echo foo bar baz quux xyxxy thud | tr ' ' '\n' | pr -2 -a -s' ' -t
foo bar
baz quux
xyxxy thud
%

Options may vary between systems; see your pr man page.

I’ve made no attempt to measure the performance of this.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .