40

Our servers are running Ubuntu Linux, and the binary file is a BSON dump of a large MongoDB collection. How reliable is a tool like split? Is there a faster or better way to do this?

5 Answers 5

41

split is very reliable. We use it for porting large log files, and it worked well for up to a couple of GBs (not 50 gb anyway).

I believe you can try using the split for your requirement, and let us know.

Split into 5GB files

split --bytes=5G inputfile

It will split into multiple files of 5GB and name it as xaa, xab, xac, .... and so on.

Concatenate

cat x* > outfile

by this you can concatenate as single file in the other end.

6
  • 1
    On OS X I had to use split -b50m to make 50 megabyte files. Note no equals sign, note lowercase.
    – funroll
    Jul 17, 2014 at 20:06
  • On OS X I have to give explicitly the output file name: split -b3m file.zip file.zip otherwise split creates files called xaa, xab, etc.
    – yannis
    Jul 3, 2020 at 13:22
  • On macOS 11.6.3, I did split -b 4000000k /path/to/input_file Feb 8, 2022 at 19:56
  • My edit attempt to change the quotes to code (commands) was rejected. If someone could kindly do that this answer would look prettier. Thanks :) Oct 10, 2022 at 5:12
  • 1
    I'd rather see split --bytes=5G inputfile and cat x* > outfile as code than as quotes. Oct 10, 2022 at 5:21
34

To split, split -b

To join, just cat.

AFAIK they are completely reliable, and I doubt there is something more efficient.

6

split & cat are totally reliable. You can additionally compress in-line like this. Suppose your input file is dump.bson:

gzip < dump.bson | split -b 32M - dump.bson.gz.

And then reconstitute with this:

cat dump.bson.gz.* | gunzip > dump.bson

Tip, this works just as well with xz(dec) in place of g(un)zip

4

If you have rar installed, it's worked very well for me:

To Separate

rar a -m0 -v5000m newfilename giantfile.foo
  • a = add files to archive
  • m0 = no compression
  • v5000m = split into chunks of 5000 megabytes

To Reassemble

Start with the first part. The rest will be found automatically:

unrar x newfilename.part1.tar
  • x = extract

Benefits:

  • CRC on the content of the split archive,
  • split-file ordering kept automatically,
  • multiple files and dirs can be included.
0
1

From your question one could assume, that between the split phase and the reassemble phase there might be a transport phase - in this case a process, that involves compression could be quite beneficial.

The standard GNU tar has the -M option to create multi-volume archives and the -L option to define the length of a segment. Together with one of the compression options (e.g. -z,j, ...) you could create a toolchain, that

  • comes with every Linux distro I know of
  • provides splitting, reassembly, compression and decompression
  • has the additional benefit of being able to pack more than one file into a single archive sequence (e.g. the MongoDB data folder as-is)

You must log in to answer this question.