I'm sure someone has had the below need, what is a quick way of splitting a huge .gz file by line? The underlying text file has 120million rows. I don't have enough disk space to gunzip the entire file at once so I was wondering if someone knows of a bash/perl script or tool that could split the file (either the .gz or inner .txt) into 3x 40mn line files. ie calling it like:
bash splitter.sh hugefile.txt.gz 4000000 1 would get lines 1 to 40 mn bash splitter.sh hugefile.txt.gz 4000000 2 would get lines 40mn to 80 mn bash splitter.sh hugefile.txt.gz 4000000 3 would get lines 80mn to 120 mn
Is perhaps doing a series of these a solution or would the gunzip -c require enough space for the entire file to be unzipped(ie the original problem): gunzip -c hugefile.txt.gz | head 4000000
Note: I can't get extra disk.