5

I'm looking for a way to compress a large file (~10GB) into several files that wont exceed 150MB each.

Any thoughts?

migrated from stackoverflow.com Aug 18 '11 at 11:14

This question came from our site for professional and enthusiast programmers.

  • how about gzip. like you tagged question with!?! – Mitch Wheat Aug 18 '11 at 10:08
10

Compress single file

This will compress file /path/to/your/large/file and creates many files with the prefix compressed.gz in the current directory, each file with a maximum size of 150000000 bytes:

gzip -c /path/to/your/large/file | split -b 150000000 - compressed.gz

Uncompress single file

To uncompress the file resulting in the uncompressed file "/path/to/decrompressed/file" compressed using the command above use:

cat compressed.gz* | zcat > /path/to/decrompressed/file
3

split [OPTION] [INPUT [PREFIX] - split a file into pieces

Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default size is 1000 lines, and default PREFIX is 'x'. With no INPUT, or when INPUT is -, read standard input.

SIZE may have a multiplier suffix: b for 512, k for 1K, m for 1 Meg.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.