I often want to archive a lot of files but haven't found a good way of doing so. There are files I packed as tar balls, some I packed as tar.gz (when compression is needed which it usually isn't for me (to be more precise: it's already compressed on the file level)), some old ones still using 7zip, some managed with git and then tarred, and even some old-fashioned zip files. None of these works well.
Currently I want to archive about 50'000 files with about 200 GB in total. These files already are compressed, so I don't need any more compression.
I had some weird things going on with 7zip (trying to remove a folder from an archive, folder is still there, archive still has the same size; trying to add a folder to the archive, archive unchanged) which is why I don't trust 7zip any more.
tar works fine but it's unnerving that the entire archive has to be read before files can be accessed.
Sometimes I need encryption, so I wrap a tar ball around the files I want to archive, sha256sum it, and store the tar ball along with a text file containing the sha256sum of the tar ball in a 7zip archive, which means that I might just have combined both issues.
zip doesn't have good encryption, so I don't want to use it.
Which method of archiving should I use? Most of the time I don't need encryption but just don't want hundreds of thousands of files lying on a hard drive (which makes keeping a checksum of everything hard (which I want in case my hard drive has some issues)) and make sure my data stays intact without having to read a tar ball for several hours when opening a 1 TB archive just to get a single file.
Operating system: Linux (Ubuntu). The distro my change but I'm pretty sure I'll use Linux when I want to access the files in the future, so support for different operating systems isn't needed.