1

I have a large text.gz file (2GB) and I want to split it by line. I tried to use:

zcat text.gz | split -l1000000

However this generated a huge file (around 92GB before I terminated it) and put too much strain on the HDD. It is possible to pipe it into gzip on the fly?

2 Answers 2

0

That can be done best with a small perl program.

I have knocked one up here: ftp://ftp.sqsol.co.uk/pub/tools/zsplit/

Take a look at it and feel free to modify it to your personal needs.

1
  • 1
    Link doesn't work, can you please post the source
    – Mark
    Nov 11, 2011 at 2:18
2

Here's a loop around awk and gzip that will split a file on line boundaries and compress the parts as it goes:

# Generate files part0.dat.gz, part1.dat.gz, etc.
prefix="part"
count=0
suffix=".dat"

lines=10000 # Split every 10000 line.

zcat thefile.dat.gz |
while true; do
  partname=${prefix}${count}${suffix}

  # Use awk to read the required number of lines from the input stream.
  awk -v lines=${lines} 'NR <= lines {print} NR == lines {exit}' >${partname}

  if [[ -s ${partname} ]]; then
    # Compress this part file.
    gzip --best ${partname}
    (( ++count ))
  else
    # Last file generated is empty, delete it.
    rm -f ${partname}
    break
  fi
done

To recreate the original file, just zcat part*.dat.gz | gzip --best >thefile1.dat.gz. The compressed file might have a different MD5 checksum from the original due to varying gzip compression options used, but the uncompressed files will be absolutely identical.

1
  • This doesn't seem to work using awk or head -n $lines because these close their stdin, I can't read lines after that. What does seem to work is using read line ; echo "$line" >> $partname the appropriate number of times
    – pascal
    Jul 30, 2014 at 14:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .