Recently I've setup a new system to handle my data hoarding needs. I have a few HDDs (2TB, 4TB, 8TB, looking to add more soon) in a StableBit DrivePool without duplication. An external 8TB HDD where I save backups with Macrium Reflect, with compression and encryption, of the most important data in the pool (will move this offsite). And I have another 8TB HDD to store SnapRAID parity for all the drives in the pool, to provide redundancy for up to one HDD failure, and data integrity with scrubbing.

Some of the data in the pool (~4TB) is old archive data that doesn't change anymore, and is occasionally added to. This is most of the stuff I backup with Macrium Reflect in the external 8TB HDD. The issue I'm facing is that this data largely consists of a lot of small files, ~2 million files in 2TB, for example. This makes most operations (re-balancing the pool, backing up with Macrium Reflect, syncing with SnapRAID, etc.) very slow, inefficient, and prone to errors (for instance if antivirus blocks some odd file then SnapRAID fails - I've yet to complete the first full sync).

So the solution I'm looking to implement is to store all these small files in a container (e.g. .tar, .zip, .7z, etc.), without encryption or compression. I think this should alleviate most of the issues by greatly reducing the number of files. So my question is what container is best for this task. I'm not looking to add redundancy, that's more flexibly handled by SnapRAID or duplication in the pool, so not looking at par/par2. But I'm looking to increase the risk of failure as little as possible should things go wrong. Particularly I'm looking for the container that is most resilient to data corruption. For example, if a container stores metadata per-file in a stream - so that one bit of corruption would only affect one file and not prevent extraction of the rest of the data - that would be preferred over a container where corruption of the header/metadata could render the entire container unusable.

Any other comments/suggestions on my proposed solution, or my system as a whole, are welcome.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.