1

I have a very large file, e.g. 100GB, and I want to compress it to many smaller zip file, with each file cannot larger than 1GB, what would be the most suitable commands to do it?

remark: I want to optimize the smaller number of files created

Update: I want each file can be decompressed independently

Improve this question
1
  • try 7-zip.org it gives better compression usually but takes lot of time and I use its GUI (linux) to split files to smaller sizes.
    – vedic
    May 22, 2014 at 5:26

1 Answer 1

Reset to default
2

Just make one large zip file and then use the linux split command to break it up into smaller pieces. cat can be used, then, to re-assemble them into a large file.

If you wanted to break up a file into 1GB chunks, the split command would be something like:

$ split -b 1073741824 myfile.zip myfile-split-

To re-assemble:

$ cat myfile-split-* > myfile.zip

Your compression ratio would drastically decrease if you actually went to multiple zip files.

I would highly recommend making sure you take an md5sum of the file before the split and then making sure the md5 matches after re-assembly.

Improve this answer
5
  • Thanks, I have updated my question, I want each file can be decompressed independently
    – Yoga
    May 22, 2014 at 5:25
  • You realize that is going to kill your compression ratio, right?
    – EEAA
    May 22, 2014 at 5:26
  • Yes, but the size limit of 1GB is a hard limit we need to follow
    – Yoga
    May 22, 2014 at 5:28
  • Why do you need each file to be able to be decompressed independently? That seems like an odd requirement.
    – EEAA
    May 22, 2014 at 5:29
  • Because this is our workflow, a program read from a compressed file and do the job, the file limit is 1GB compressed
    – Yoga
    May 22, 2014 at 5:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .