Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to process a fairly large tab-separated file (~30GB). I can run awk to rearrange the columns, but it only uses a single core of my 8 core machine. How can I use all the cores? (and dramatically reduce my processing time) without physically splitting the file and using up more disk space?

cut -f3,4,5,6 original.tsv | awk 'BEGIN { FS="\t" }; { print $2"\t"$1"\t"$3"\t"$4 }' > abridged.tsv

I would like to do something like this, but without using up more disk space:

split -l 50000 original.tsv sp
for i in $(dir sp*); do ./ $i & done;

where is essentially the cut/awk statement above.

Again, the key problem here is to do this process without using another 30GB! Any suggestions?

share|improve this question
Are you sure the CPU is the bottleneck, not the IO? – Marco Jul 18 '12 at 9:29
You don't need the cut command just print $4,$3,$5,$6 (and set OFS to tab) – RedGrittyBrick Jul 18 '12 at 10:06
1. I/O is most certainly the bottleneck, not cores. 2. It is not a good parallelizable task anyway unless every row is a defined number of bits long. Since you also want in-place editing, I'd suggest to go to Perl for the operation. In-place editing is somewhat possible with awk and some redirection trickery, but Perl solutions are more robust in this aspect, I feel. Also, while testing the solution out, make sure you have a backup of the original file (even if it is 30GB) or you might be sorry. – Daniel Andersson Jul 18 '12 at 11:07
I don't know much about awk (and therefore can't say much about it) but I thought it might enlighten your discussion if I threw this into your discussion. – wullxz Jul 27 '12 at 14:48

Since you are using "split" to separate your files by number of lines, then process them separately outputting to different files (I guess), you could do several "awk" commands each processing only part of your file based on the line number:

$ cut -f3,4,5,6 original.tsv | awk 'BEGIN { FS="\t" }; (NR < 50000){ print $2"\t"$1"\t"$3"\t"$4 }' > abridged01.tsv
$ cut -f3,4,5,6 original.tsv | awk 'BEGIN { FS="\t" }; ((NR >= 50000) && (NR < 100000)){ print $2"\t"$1"\t"$3"\t"$4 }' > abridged02.tsv
$ cut -f3,4,5,6 original.tsv | awk 'BEGIN { FS="\t" }; ((NR >= 100000) && (NR < 150000)){ print $2"\t"$1"\t"$3"\t"$4 }' > abridged03.tsv

NR is an internal variable to "awk" that contains the current line number. Each command would process only the lines in their range. BUT, they would all go through the other lines also, because they need to count them. I'm pretty sure it won't help you, because you most probably will fall in the IO bottleneck. But you will have multiple processes, allowing you to use multiple CPUs, if that's what you want. ;-)

Now, IF you have all lines with the same length in bytes, you surely can do a real parallelization. In this case, you will use "dd" to extract the exact portion for each "awk" process. You would do something similar to:

dd if=original.tsv bs=30 count=50000 skip=0 | cut -f3,4,5,6 | awk 'BEGIN { FS="\t" }; { print $2"\t"$1"\t"$3"\t"$4 }' > abridged01.tsv

dd if=original.tsv bs=30 count=50000 skip=50000 | cut -f3,4,5,6 | awk 'BEGIN { FS="\t" }; { print $2"\t"$1"\t"$3"\t"$4 }' > abridged02.tsv

dd if=original.tsv bs=30 count=50000 skip=100000 | cut -f3,4,5,6 | awk 'BEGIN { FS="\t" }; { print $2"\t"$1"\t"$3"\t"$4 }' > abridged03.tsv

Where 30 is the number of bytes in each line. If your lines are not all the same size in bytes (which is the most probable) but you know the exact byte where your block of lines start and end, you can still do it using dd. Study its parameters. Finally, if you don't know where the blocks start and end, you can find them out with an extra awk command. But that adds an extra full read in your file. Unless you will process your original.tsv file multiple times in different ways, you will certainly spend more time pre-processing (calculating the bytes where line blocks start and end) and then processing (which will probably have a small gain because you will certainly have an IO bottleneck) than if you simply used the solution you already know.

In any case, now you have information and options. ;-) Good luck! (y)

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .