Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm sure someone has had the below need, what is a quick way of splitting a huge .gz file by line? The underlying text file has 120million rows. I don't have enough disk space to gunzip the entire file at once so I was wondering if someone knows of a bash/perl script or tool that could split the file (either the .gz or inner .txt) into 3x 40mn line files. ie calling it like:

    bash hugefile.txt.gz 4000000 1
 would get lines 1 to 40 mn    
    bash hugefile.txt.gz 4000000 2
would get lines 40mn to 80 mn
    bash hugefile.txt.gz 4000000 3
would get lines 80mn to 120 mn

Is perhaps doing a series of these a solution or would the gunzip -c require enough space for the entire file to be unzipped(ie the original problem): gunzip -c hugefile.txt.gz | head 4000000

Note: I can't get extra disk.


share|improve this question

migrated from Jan 23 '12 at 11:28

This question came from our site for professional and enthusiast programmers.

Do you want the resulting files to be gziped again? – user95605 Jan 23 '12 at 11:25
You can use gunzip in a ipe. The rest can be done with head and tail – Ingo Jan 23 '12 at 11:25
@Tichodroma - no I don't need them gziped again. But I could not store all the split text files at once. So i would like to get the first split, do stuff with it, then delete the first split, and then get the 2nd split.etc finally removing the original gz – toop Jan 23 '12 at 11:42
@toop: Thanks for the clarification. Note that it's generally better to edit your question if you want to clarify it, rather than put it into a comment; that way everyone will see it. – sleske Jan 23 '12 at 12:06
The accepted answer is good if you only want a fraction of the chunks, and do not know them in advance. If you want to generate all the chunks at once, the solutions based on split will be a lot faster ,O(N) instead of O(N²). – b0fh Aug 13 '14 at 15:27
up vote 7 down vote accepted

Yes, using

gunzip -c hugefile.txt.gz | head -n 4000000

is one way of tackling this problem (note the "-n", that was missing in your question). That would output the first 4000000 lines on standard out - you probably want to append another pipe to actually do something with the data.

To get the other parts, you'd use a combination of head and tail, like:

gunzip -c hugefile.txt.gz | head -n 8000000 |tail -n 4000000

to get the second block.

Is perhaps doing a series of these a solution or would the gunzip -c require enough space for the entire file to be unzipped

No, the gunzip -c does not require any disk space - it does everything in memory, then streams it out to stdout.

share|improve this answer
From performance view: does gzip actually unzip whole file? Or is it able to "magically" know that only 4mn lines are needed? – Alois Mahdal Mar 22 '12 at 12:57
@AloisMahdal: Actually, that would be a good separate question :-). Short version: gzip does not know about the limit (which comes from a different process). If head is used, head will exit when it has received enough, and this will propagate to gzip (via SIGPIPE, see Wikipedia). For tail this is not possible, so yes, gzip will decompress everything. – sleske Mar 22 '12 at 15:26
But if you are interested, you should really ask this as a separate question. – sleske Mar 22 '12 at 15:35

As you are working on a (non-rewindable) stream, you will want to use the '+N' form of tail to get lines starting from line N onwards.

zcat hugefile.txt.gz | head -n 40000000
zcat hugefile.txt.gz | tail -n +40000001 | head -n 40000000
zcat hugefile.txt.gz | tail -n +80000001 | head -n 40000000
share|improve this answer

pipe to split use either gunzip -c or zcat to open the file

gunzip -c bigfile.gz | split -l 400000

Add output specifications to the split command.

share|improve this answer
This is massively more efficient than the accepted answer, unless you only require a fraction of the split chunks. Please upvote. – b0fh Aug 13 '14 at 15:29

I'd consider using split.

split a file into pieces

share|improve this answer

Here's a perl program that can be used to read stdin, and split the lines, piping each clump to a separate command that can use a shell variable $SPLIT to route it to a different destination. For your case, it would be invoked with

zcat hugefile.txt.gz | perl 40000000 'cat > tmp$SPLIT.txt; do_something tmp$SPLIT.txt; rm tmp$SPLIT.txt'

Sorry the command-line processing is a little kludgy but you get the idea.

#!/usr/bin/perl -w
# like xargs but instead of clumping input into each command's args, clumps it into each command's input.
# Usage: perl LINES 'COMMAND'
# where: 'COMMAND' can include shell variable expansions and can use $SPLIT, e.g.
#   'cat > tmp$SPLIT.txt'
# or:
#   'gzip > tmp$SPLIT.gz'
use strict;

sub pipeHandler {
    my $sig = shift @_;
    print " Caught SIGPIPE: $sig\n";
$SIG{PIPE} = \&pipeHandler;

my $LINES = shift;
die "LINES must be a positive number\n" if ($LINES <= 0);
my $COMMAND = shift || die "second argument should be COMMAND\n";

my $line_number = 0;

while (<STDIN>) {
    if ($line_number%$LINES == 0) {
        close OUTFILE;
        my $split = $ENV{SPLIT} = sprintf("%05d", $line_number/$LINES+1);
        print "$split\n";
        my $command = $COMMAND;
        open (OUTFILE, "| $command") or die "failed to write to command '$command'\n";
    print OUTFILE $_;

exit 0;
share|improve this answer

Here's a python script to open a globbed set of files from a directory, gunzip them if necessary, and read through them line by line. It only uses the space necessary in memory for holding the filenames, and the current line, plus a little overhead.

#!/usr/bin/env python
import gzip, bz2
import os
import fnmatch

def gen_find(filepat,top):
    for path, dirlist, filelist in os.walk(top):
        for name in fnmatch.filter(filelist,filepat):
            yield os.path.join(path,name)

def gen_open(filenames):
    for name in filenames:
        if name.endswith(".gz"):
        elif name.endswith(".bz2"):
            yield bz2.BZ2File(name)
            yield open(name)

def gen_cat(sources):
    for s in sources:
        for item in s:
            yield item

def main(regex, searchDir):
    fileNames = gen_find(regex,searchDir)
    fileHandles = gen_open(fileNames)
    fileLines = gen_cat(fileHandles)
    for line in fileLines:
        print line

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Search globbed files line by line', version='%(prog)s 1.0')
    parser.add_argument('regex', type=str, default='*', help='Regular expression')
    parser.add_argument('searchDir', , type=str, default='.', help='list of input files')
    args = parser.parse_args()
    main(args.regex, args.searchDir)

The print line command will send every line to std out, so you can redirect to a file. Alternatively, if you let us know what you want done with the lines, I can add it to the python script and you won't need to leave chunks of the file laying around.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .