I have a large (~100GB uncompressed) collection of files that I want to store in < 2GB chunks for backup on a Windows Server based storage system. I'm running Ubuntu on my workstation.
The data sets are collected in subdirectories, with a tiered type of structure. (e.g., directory A contains three subdirectories 1, 2, 3, which each contain maybe 30ish subdirectories, which each contain 6 or 7 files, with the same names (in each subdirectory))
tar
will compress and archive my data, but if I want to access individual sections, it's pretty painful. It's also slow, and if I use compression, I have to uncompress before I can extract individual directories (afaik)
This would be using something like:
tar -cf mySuperStructure.tar;
split -b 1024M mySuperStructure.tar mySuperStructure.tar.part- ;
gzip mySuperStructure.tar.part-* ;
or similar - I know tar has a -z
option, but I think this might be problematic when using split
? Likewise, I believe there is an option to split tar
archives, possibly with compression, but the files are split in unhelpful ways (Perhaps this is the best solution, if so - please advise)
Alternatively, dar
will compress on a file by file level, allowing a -m
option to specify a minimum size. This would be ideal, but for the large number of files making it take a long time to compress and archive the setup. The lowest level directories are ~70MB in size, so compressing these would be useful (and I suspect faster, since fewer compression operations?)
Can I specify the compression of the subdirectories individually, without individual files being compressed? Would this be any faster if I could?
A manual implementation might look like:
for levelA in $(ls); do
cd levelA;
for subdirectoryCase in $(ls); do
cd subdirectoryCase;
for subdirSmall in $(ls); do
gzip subdirSmall;
done
cd ..;
tar -cf $subdirectoryCase.tar $subdirectoryCase;
gzip $subdirectoryCase.tar;
done
cd ..;
tar -cf $levelA.tar $levelA;
gzip $levelA.tar;
done
tar -cf superStructure.tar levelA1.tar levelA2.tar levelA3.tar;
gzip superStructure.tar;
but this seems like it might be micromanaging to a horrible level (but that might still be best, I don't know?) dar
could be substituted for tar
throughout, and split
could be used perhaps before the final gzip
, assuming that the earlier gzip'd files were sufficiently small (which they won't be, but that's presumably possible to deal with in a similar manner)