Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a large text.gz file (2GB) and I want to split it by line. I tried to use:

zcat text.gz | split -l1000000

However this generated a huge file (around 92GB before I terminated it) and put too much strain on the HDD. It is possible to pipe it into gzip on the fly?

share|improve this question

2 Answers 2

up vote 0 down vote accepted

That can be done best with a small perl program.

I have knocked one up here: ftp://ftp.sqsol.co.uk/pub/tools/zsplit/

Take a look at it and feel free to modify it to your personal needs.

share|improve this answer
    
Thanks, it's very useful. –  yumyai Apr 18 '11 at 9:31
    
Link doesn't work, can you please post the source –  Mark Nov 11 '11 at 2:18

Here's a loop around awk and gzip that will split a file on line boundaries and compress the parts as it goes:

# Generate files part0.dat.gz, part1.dat.gz, etc.
prefix="part"
count=0
suffix=".dat"

lines=10000 # Split every 10000 line.

zcat thefile.dat.gz |
while true; do
  partname=${prefix}${count}${suffix}

  # Use awk to read the required number of lines from the input stream.
  awk -v lines=${lines} 'NR <= lines {print} NR == lines {exit}' >${partname}

  if [[ -s ${partname} ]]; then
    # Compress this part file.
    gzip --best ${partname}
    (( ++count ))
  else
    # Last file generated is empty, delete it.
    rm -f ${partname}
    break
  fi
done

To recreate the original file, just zcat part*.dat.gz | gzip --best >thefile1.dat.gz. The compressed file might have a different MD5 checksum from the original due to varying gzip compression options used, but the uncompressed files will be absolutely identical.

share|improve this answer
    
This doesn't seem to work using awk or head -n $lines because these close their stdin, I can't read lines after that. What does seem to work is using read line ; echo "$line" >> $partname the appropriate number of times –  pascal Jul 30 at 14:11

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.