0

I often want to archive a lot of files but haven't found a good way of doing so. There are files I packed as tar balls, some I packed as tar.gz (when compression is needed which it usually isn't for me (to be more precise: it's already compressed on the file level)), some old ones still using 7zip, some managed with git and then tarred, and even some old-fashioned zip files. None of these works well.

Currently I want to archive about 50'000 files with about 200 GB in total. These files already are compressed, so I don't need any more compression.

I had some weird things going on with 7zip (trying to remove a folder from an archive, folder is still there, archive still has the same size; trying to add a folder to the archive, archive unchanged) which is why I don't trust 7zip any more.

tar works fine but it's unnerving that the entire archive has to be read before files can be accessed.

Sometimes I need encryption, so I wrap a tar ball around the files I want to archive, sha256sum it, and store the tar ball along with a text file containing the sha256sum of the tar ball in a 7zip archive, which means that I might just have combined both issues.

zip doesn't have good encryption, so I don't want to use it.

Which method of archiving should I use? Most of the time I don't need encryption but just don't want hundreds of thousands of files lying on a hard drive (which makes keeping a checksum of everything hard (which I want in case my hard drive has some issues)) and make sure my data stays intact without having to read a tar ball for several hours when opening a 1 TB archive just to get a single file.

Operating system: Linux (Ubuntu). The distro my change but I'm pretty sure I'll use Linux when I want to access the files in the future, so support for different operating systems isn't needed.

  • 2
    What exactly do you mean by 'archiving'? Why do you have to use tar/zip at all? Just copy them to a spare drive and be done with it? – Zoredache Jan 20 '16 at 18:04
  • Exactly what Zoredache said, archive files by just copying them to some other drive/partition/disk/dvd/etc. The point of a backup is to keep in case there is a media failure or accidental deletion (or malware / ransomware encryption/deletion), keeping hashes is fine, and multiple backups is even better... but why bother with a giant tar if you're not even using compression? And encryption isn't a tar ball with a sha256 hash, I think 7zip keeps it's own hashes, and if you're using 7zip's encryption it's better than nothing but nowhere near equal to gpg. – Xen2050 Jan 21 '16 at 0:00
  • I don't want to keep millions of files. It's hard to keep track of them and I'm using characters in file names not every file system can cope with but want to be able to give archives to others so they are stored in several locations. That's where a single file per archive is handy. 7zip is very good at breaking things, not in storing things. I never claimed creating a sha256sum of something magically encrypts it. – UTF-8 Jan 21 '16 at 0:42
  • Should I just create a file containing an ext4 file system holding my files? There got to be a better solution. – UTF-8 Jan 21 '16 at 0:47
0

This is the closest to an answer I can think of:

Use Backup Software for Ubuntu (and any Ubuntu-derivatives, like Linux Mint. Most should work for any Debian and just about any Linux too). Don't re-invent the wheel (unless you'd really like to).

Here's some excellent resources:

I'd probably pick one of the popular answers from AskUbuntu.com, if I didn't want to just copy files to an optionally encrypted partition.

And this ArchLinux wiki / help page on Disk Encryption is especially excellent, Read it!

  • I don't want to back stuff up. Actually, I do. I have a working solution backing up everything in my home folder every 2 hours or 20 minutes after logging into my machine. But that's not what I'm asking about. I'm asking about files which I'll probably not use a lot but want to make very sure they exist in the future. – UTF-8 Jan 21 '16 at 0:45
  • You want to copy files, right? Backup means: a copy or duplicate version, especially of a file, program, or entire computer system, retained for use in the event that the original is in some way rendered unusable. You should probably be making multiple DVD type backup copies, they seem to be the most stable, I've read they should last for at least 5 years, possibly 10 or 20, maybe even 70+ – Xen2050 Jan 21 '16 at 3:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.