Our servers are running Ubuntu Linux, and the binary file is a BSON dump of a large MongoDB collection. How reliable is a tool like split
? Is there a faster or better way to do this?
5 Answers
split
is very reliable. We use it for porting large log files, and it worked well for up to a couple of GBs (not 50 gb anyway).
I believe you can try using the split for your requirement, and let us know.
Split into 5GB files
split --bytes=5G inputfile
It will split into multiple files of 5GB and name it as xaa, xab, xac, .... and so on.
Concatenate
cat x* > outfile
by this you can concatenate as single file in the other end.
-
1On OS X I had to use split -b50m to make 50 megabyte files. Note no equals sign, note lowercase.– funrollJul 17, 2014 at 20:06
-
On OS X I have to give explicitly the output file name:
split -b3m file.zip file.zip
otherwisesplit
creates files calledxaa
,xab
, etc.– yannisJul 3, 2020 at 13:22 -
-
My edit attempt to change the quotes to code (commands) was rejected. If someone could kindly do that this answer would look prettier. Thanks :) Oct 10, 2022 at 5:12
-
1I'd rather see
split --bytes=5G inputfile
andcat x* > outfile
as code than as quotes. Oct 10, 2022 at 5:21
To split, split -b
To join, just cat
.
AFAIK they are completely reliable, and I doubt there is something more efficient.
split
& cat
are totally reliable. You can additionally compress in-line like this. Suppose your input file is dump.bson
:
gzip < dump.bson | split -b 32M - dump.bson.gz.
And then reconstitute with this:
cat dump.bson.gz.* | gunzip > dump.bson
Tip, this works just as well with xz(dec)
in place of g(un)zip
If you have rar installed, it's worked very well for me:
To Separate
rar a -m0 -v5000m newfilename giantfile.foo
- a = add files to archive
- m0 = no compression
- v5000m = split into chunks of 5000 megabytes
To Reassemble
Start with the first part. The rest will be found automatically:
unrar x newfilename.part1.tar
- x = extract
Benefits:
- CRC on the content of the split archive,
- split-file ordering kept automatically,
- multiple files and dirs can be included.
From your question one could assume, that between the split phase and the reassemble phase there might be a transport phase - in this case a process, that involves compression could be quite beneficial.
The standard GNU tar
has the -M
option to create multi-volume archives and the -L
option to define the length of a segment. Together with one of the compression options (e.g. -z
,j
, ...) you could create a toolchain, that
- comes with every Linux distro I know of
- provides splitting, reassembly, compression and decompression
- has the additional benefit of being able to pack more than one file into a single archive sequence (e.g. the MongoDB data folder as-is)