Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Running Linux. I have a directory of around 150 large CSV files; simply doing a zip -9 on them results in a monolithic file that is still too large. I would like it to simply zip them in maybe four or five zip files of 30-40 CSVs each; this way sequencing or spanned zip order won't be a problem, as each zip is independent. There must be a simple way to do this. Any suggestions?

(and yes, zip is the preferred format, if possible)

share|improve this question

migrated from stackoverflow.com Jun 2 '13 at 7:27

This question came from our site for professional and enthusiast programmers.

2 Answers 2

Isn't -s switch enough? You may use zip -s to split the file into files of maximum size, e.g.:

"zip -s 300m <2 gb file>" produces:

file.zip (300 mb, master file)
file.001.zip (300 mb)
file.002.zip (300 mb)
file.003.zip (300 mb)
file.004.zip (300 mb)
file.005.zip (300 mb)
file.006.zip (200 mb)

Then "unzip file.zip" will unzip everything together.

share|improve this answer
    
What version of zip is this?? I get file.z01 file.z02 ... file.zip and unzip file.zip does not work directly (I would use zip -F to recombine them first). Note these are not "independent" as requested. –  sourcejedi Jun 2 '13 at 10:22
1  
@sourcejedi: In this answer (superuser.com/a/602736/195224) are some more detailed explanations. –  mpy Jun 2 '13 at 10:43
    
@mpy I know, I've just written that answer :). –  sourcejedi Jun 2 '13 at 11:44
    
@sourcejedi: Oh yes, now you say it... ;) –  mpy Jun 2 '13 at 11:52

Use split on the list of input files :-).

(Not tested, I've included rm commands for cleanup, take care).

ls *.csv > csvfiles
split -d -l30 - csvfiles < csvfiles
for i in csvfiles[0-9][0-9]; do
  zip "$i.zip" -@ < "$i"
done

rm csvfiles
rm csvfiles[0-9][0-9]
share|improve this answer
    
Why do you use split -C (--line-bytes) and not split -l (--lines)? That would be more predictable, with regard to how many CSV files are in one archive. –  mpy Jun 2 '13 at 12:00
    
I skimmed the manpage too quickly. Thanks, I'll fix it! –  sourcejedi Jun 2 '13 at 12:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.