1

I have a large (~100GB uncompressed) collection of files that I want to store in < 2GB chunks for backup on a Windows Server based storage system. I'm running Ubuntu on my workstation.

The data sets are collected in subdirectories, with a tiered type of structure. (e.g., directory A contains three subdirectories 1, 2, 3, which each contain maybe 30ish subdirectories, which each contain 6 or 7 files, with the same names (in each subdirectory))

tar will compress and archive my data, but if I want to access individual sections, it's pretty painful. It's also slow, and if I use compression, I have to uncompress before I can extract individual directories (afaik)

This would be using something like:

tar -cf mySuperStructure.tar;
split -b 1024M mySuperStructure.tar mySuperStructure.tar.part- ;
gzip mySuperStructure.tar.part-* ;

or similar - I know tar has a -z option, but I think this might be problematic when using split? Likewise, I believe there is an option to split tar archives, possibly with compression, but the files are split in unhelpful ways (Perhaps this is the best solution, if so - please advise)

Alternatively, dar will compress on a file by file level, allowing a -m option to specify a minimum size. This would be ideal, but for the large number of files making it take a long time to compress and archive the setup. The lowest level directories are ~70MB in size, so compressing these would be useful (and I suspect faster, since fewer compression operations?)

Can I specify the compression of the subdirectories individually, without individual files being compressed? Would this be any faster if I could?

A manual implementation might look like:

for levelA in $(ls); do
cd levelA;
    for subdirectoryCase in $(ls); do
    cd subdirectoryCase;
        for subdirSmall in $(ls); do
        gzip subdirSmall;
        done
    cd ..;
    tar -cf $subdirectoryCase.tar $subdirectoryCase;
    gzip $subdirectoryCase.tar;
    done
cd ..;
tar -cf $levelA.tar $levelA;
gzip $levelA.tar;
done

tar -cf superStructure.tar levelA1.tar levelA2.tar levelA3.tar;
gzip superStructure.tar;

but this seems like it might be micromanaging to a horrible level (but that might still be best, I don't know?) dar could be substituted for tar throughout, and split could be used perhaps before the final gzip, assuming that the earlier gzip'd files were sufficiently small (which they won't be, but that's presumably possible to deal with in a similar manner)

3
  • why not store them on a Windows compressed volume without archiving?
    – Keltari
    Aug 27, 2014 at 2:22
  • @Keltari - not sure I understand? I'm assuming from the words in your sentence you mean something like, place the files, then squash them, but I can only upload individual files to the server, and I don't know about 'compressed volumes'. I really have no idea about Windows filesystems or storage :/
    – chrisb2244
    Aug 27, 2014 at 2:34
  • If you just want to explore and micromanage your files inside archives, maybe AVFS may interest you?
    – gaborous
    Mar 9, 2015 at 22:05

0

You must log in to answer this question.

Browse other questions tagged .