22

Our servers are running Ubuntu Linux, and the binary file is a BSON dump of a large MongoDB collection. How reliable is a tool like split? Is there a faster or better way to do this?

migrated from stackoverflow.com Jul 6 '10 at 1:58

This question came from our site for professional and enthusiast programmers.

23

To split, split -b

To join, just cat.

AFAIK they are completely reliable, and I doubt there is something more efficient.

17

split is very reliable. We use it for porting large log files, and it worked well for up to a couple of GBs ( not 50 gb anyway ).

I believe you can try using the split for your requirement, and let us know.

Split into 5GB files

split --bytes=5G inputfile

It will split into multiple files of 5GB and name it as xaa, xab, xac, .... and so on.

Concatenate

cat x* > outfile

by this you can concatenate as single file in the other end.

  • 1
    On OS X I had to use split -b50m to make 50 megabyte files. Note no equals sign, note lowercase. – funroll Jul 17 '14 at 20:06
3

If you have rar installed, it's worked very well for me:

To Separate

rar a -m0 -v5000m newfilename giantfile.foo
  • a = add files to archive
  • m0 = no compression
  • v5000m = split into chunks of 5000 megabytes

To Reassemble:

unrar x newfilename.*
  • x = extract

Benefits:

  • CRC on the content of the split archive,
  • split-file ordering kept automatically,
  • multiple files and dirs can be included.
3

split & cat are totally reliable. You can additionally compress in-line like this. Suppose your input file is dump.bson:

gzip < dump.bson | split -b 32M - dump.bson.gz.

And then reconstitute with this:

cat dump.bson.gz.* | gunzip > dump.bson

Tip, this works just as well with xz(dec) in place of g(un)zip

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.