Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a very large file, e.g. 100GB, and I want to compress it to many smaller zip file, with each file cannot larger than 1GB, what would be the most suitable commands to do it?

remark: I want to optimize the smaller number of files created

Update: I want each file can be decompressed independently

share|improve this question

migrated from serverfault.com May 22 '14 at 7:27

This question came from our site for system and network administrators.

    
try 7-zip.org it gives better compression usually but takes lot of time and I use its GUI (linux) to split files to smaller sizes. – vedic May 22 '14 at 5:26

Just make one large zip file and then use the linux split command to break it up into smaller pieces. cat can be used, then, to re-assemble them into a large file.

If you wanted to break up a file into 1GB chunks, the split command would be something like:

$ split -b 1073741824 myfile.zip myfile-split-

To re-assemble:

$ cat myfile-split-* > myfile.zip

Your compression ratio would drastically decrease if you actually went to multiple zip files.

I would highly recommend making sure you take an md5sum of the file before the split and then making sure the md5 matches after re-assembly.

share|improve this answer
    
Thanks, I have updated my question, I want each file can be decompressed independently – Yoga May 22 '14 at 5:25
    
You realize that is going to kill your compression ratio, right? – EEAA May 22 '14 at 5:26
    
Yes, but the size limit of 1GB is a hard limit we need to follow – Yoga May 22 '14 at 5:28
    
Why do you need each file to be able to be decompressed independently? That seems like an odd requirement. – EEAA May 22 '14 at 5:29
    
Because this is our workflow, a program read from a compressed file and do the job, the file limit is 1GB compressed – Yoga May 22 '14 at 5:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .