Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Consider a stream of one-word lines such as the stdout of

$ echo foo bar baz quux xyxxy thud | tr ' ' '\n'
foo
bar
baz
quux
xyxxy
thud

I say "one-word" to indicate there is no whitespace other than the newlines. I'd like to "columnize" the data by filling in a fixed number of fields (not single characters) per line, one row at a time. For small amounts of data, xargs does this nicely:

$ echo foo bar baz quux xyxxy thud | tr ' ' '\n' | xargs -n 2
foo bar
baz quux
xyxxy thud

However using xargs this way is slow. I have hundreds of gigabytes of text to process, so I would very much like my write throughput to hit the 50 MB/s or so that I know this hardware can do (the xargs above is orders of magnitude slower.)

Is there something better available in the standard toolbox? I can write something if needed but I'd prefer using a text filter that's already there.

share|improve this question

2 Answers 2

up vote 0 down vote accepted

The best I can come up with is

$ echo foo bar baz quux xyxxy thud | tr ' ' '\n' \
  | perl -lne '$x.=" $_"; if(!($. % 2)){print $x; $x="";} END{print $x if $x}'
 foo bar
 baz quux
 xyxxy thud

or the arguably more elegant, though no shorter

$ echo foo bar baz quux xyxxy thud bang| tr ' ' '\n' \
  | perl -ne 'chomp; print "$_ "; print "\n" if (!($. % 3)); END {print "\n"}'
foo bar baz
quux xyxxy thud
bang

Efficiency?

$ time perl -e 'print "foo\n" for (1..10000)' | xargs -n 3 > /dev/null

real    0m1.330s
user    0m0.500s
sys     0m0.830s


$ time perl -e 'print "foo\n" for (1..10000)' \
  | perl -ne 'chomp; print "$_ "; print "\n" if (!($. % 3)); END {print "\n"}' \ 
  > /dev/null

real    0m0.060s
user    0m0.030s
sys     0m0.030s

Initially I tried 1000000 words but got fed up waiting for xargs though perl was 1.45s

(Timings done on andLinux under Vista-32 on an AMD-64 x2 5600+ 2.8GHz with 4GB RAM)

share|improve this answer
    
The perl one-liner hits about 20 MB/s write to disk (on my hardware), but there's enough CPU left that I can pipeline them. Thanks! –  phs Sep 3 '11 at 19:25

pr is vastly underrated.  Here’s what I came up with:

% echo foo bar baz quux xyxxy thud | tr ' ' '\n' | pr -2 -a -s' ' -t
foo bar
baz quux
xyxxy thud
%

Options may vary between systems; see your pr man page.

I’ve made no attempt to measure the performance of this.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.