How to verify a tar file in Windows?

Question

In an effort to create more disk space I am trying to compress a bunch of my data into .tar.gz files. This takes a long time, so I'd like to automate the process in a .bat file. Ideally I would create the .tar.gz file, verify that it is a valid file, and then delete the uncompressed data from the server. How do I verify that the new tar file is valid in Windows? The Linux version of tar has the -W option which seems perfect, but that isn't included in the Windows version. I've been manually running tar -t and verifying that it didn't encounter any errors before deleting the original data, but I'm not sure how I'd use that in a script.

Answer 1

Generally, it's unlikely, unless you have a serious problem with your RAM or storage, that an archive that you just created is broken. What should have happened? tar and consorts are more-than-well-tested. Also, you're asking the same program running on the same machine that you suspect might produce broken files to check whether it did. Raises a lot of questions on your mental model of what could fail!

---

So, "it's taking a long time": Let's address that first.

When doing something like

tar czf compressed.tar.gz folder1/ folder2/ 

tar internally uses a single threaded version of the gzip compressor.

Not only is gzip not a very good compressor, it's also slow, and only using a single thread doesn't help, either.

You could achieve the same result, but much faster (assuming you have more than one CPU core, but even on a single core, this might be beneficial) through

tar czf - folder1/ folder2/ | pigz > compressed.tar.gz

You can get better (but slightly slower) gzip compression using

tar czf - folder1/ folder2/ | pigz --best > compressed.tar.gz

pigz is just a multi-threaded gzip implementation.

But gzip (and thus also pigz) is a terrible compressor, as said, slow and inefficient. Use something that's not designed with 1980's PCs in mind, and you get much better speeds and/or rates. Zstd is popular for backup compression. At a speed that should be very similar to that of gzip:

tar czf - folder1/ folder2/ | zstd -10 > compressed-roughly-as-fast-as-gzip.tar.zst

You'll notice that for most types of data, the result uses about 30% less space (very roughly, really depends on the data!).

And to compress roughly equally well, but much much faster

tar czf - folder1/ folder2/ | zstd -3 > compressed-roughly-as-good-as-gzip.tar.zst

Both zstd and pigz should be available the same way you've installed your tar.

The Linux version of tar has the -W option which seems perfect, but that isn't included in the Windows version

Ah, your Linux version is probably GNU tar, whereas the "Window version" (whatever that is) probably is a different tar implementation (maybe the busybox or bsdtar version?).

---

How do I verify that the new tar file is valid in Windows?

It's darn hard to make a tar file invalid. Basically, you can truncate it.

So, an easy way of checking whether the tar file is valid is to simply decompress it. Checking whether the contents are intact is a different problem, and can essentially only be solved by extracting it and comparing the files (that's what GNU tar does with -W, but it omits the writing to a file, does the comparison in RAM).

You don't have a GNU tar with the -W option, so you can then do something like (assuming bash):

# create compressed archive
original_folder="$(pwd)"
archivename="compressed.tar.zst"
tar xvf - original_files | zstd -12 > "${archivename}"

# Verify
## enable recursive globbing
setopt globstar
## make temporary directory
tmpdir="$(mktemp -d)"
cd "${tmpdir}"
## extract
tar xf "${original_folder}/${archivename}" || echo "Decompression failed"
## go through all files, hash' em
hashes="$(sha256sum --tag **/*)"
cd "${original_folder}"
echo "${hashes}" | sha256sum --check  || echo "hash verification failed"
## Clean up
rm -rf "${tmpdir}"