Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a very large file, e.g. 100GB, and I want to compress it to many smaller zip file, with each file cannot larger than 1GB, what would be the most suitable commands to do it?

remark: I want to optimize the smaller number of files created

Update: I want each file can be decompressed independently

share|improve this question

migrated from serverfault.com May 22 at 7:27

This question came from our site for system and network administrators.

    
try 7-zip.org it gives better compression usually but takes lot of time and I use its GUI (linux) to split files to smaller sizes. –  vedic May 22 at 5:26

1 Answer 1

Just make one large zip file and then use the linux split command to break it up into smaller pieces. cat can be used, then, to re-assemble them into a large file.

If you wanted to break up a file into 1GB chunks, the split command would be something like:

$ split -b 1073741824 myfile.zip myfile-split-

To re-assemble:

$ cat myfile-split-* > myfile.zip

Your compression ratio would drastically decrease if you actually went to multiple zip files.

I would highly recommend making sure you take an md5sum of the file before the split and then making sure the md5 matches after re-assembly.

share|improve this answer
    
Thanks, I have updated my question, I want each file can be decompressed independently –  Yoga May 22 at 5:25
    
You realize that is going to kill your compression ratio, right? –  EEAA May 22 at 5:26
    
Yes, but the size limit of 1GB is a hard limit we need to follow –  Yoga May 22 at 5:28
    
Why do you need each file to be able to be decompressed independently? That seems like an odd requirement. –  EEAA May 22 at 5:29
    
Because this is our workflow, a program read from a compressed file and do the job, the file limit is 1GB compressed –  Yoga May 22 at 5:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.