3

As question above, does 7zip (more specifically p7zip on Linux) use disk space while testing archives? Since I only have a 2TB drive to work with and with every archive ranging 800GB-1TB in size, I was thinking to test 2 archives at the same time instead of just one.

The 7zip official documentation does not mention about disk usage while testing.

2 Answers 2

0

It shouldn't. (But it might)

In order to verify that the data in an archive is correct when extracting it every file or block of data will have a CRC or error detection code associated with it.

In decompressing a file it makes a lot of sense, from an efficiency perspective, to do the error checking before writing data to disk. Otherwise you are wasting precious resources reading from the archive, writing it to disk, then re-reading the on-disk data to do error checking. With a large archive or on a memory constrained system this could double the time taken to decompress a file which would be unacceptable. I'm assuming in this case that disk reading and writing are the slowest part of the process.

If you do the check before writing you can effectively stream the archive through the decompressor, through your error checking algorithm, then out to disk and assume that the disk subsystem knows what it's doing. Job done.

By doing it this way "testing" the archive becomes a free operation. You follow exactly the same steps as you would for decompression, but you just throw away the data without writing it to disk.

I strongly expect that this is the way it works, because writing everything to disk simply to test an archive seems insane and would be no faster than a "real" decompression of the data. That "testing" is faster implies that at least one step, most likely writing data to disk, is skipped.

2
  • So that means I can test multiple 7zip files at the same time? I afraid that testing multiple 7zip files simultaneously will result in corruption. Mar 20, 2017 at 3:09
  • Slowness yes, as they will all be competing for disk reading and CPU time, but there should be no corruption.
    – Mokubai
    Mar 20, 2017 at 3:44
0

No, it does not - at least not the 19.00 version.

Testing multiple files in parallel works great on solid state drives, but usually kills performance on mechanical drives - a lot of seeking is then involved to read multiple archives in parallel. Thus, the following recommendation may be made:

  • When testing archives on solid state drives (NVMe or SATA SSD), launch as many processes as you have cores (if you can).

  • When testing archives on the same mechanical drive (or mechanical RAID volume), launch one, or at most two processes.

  • When testing archives on USB drives, the results will vary.

The sad majority of common USB "pen drives" or "sticks" is abysmally slow, even when reading - i.e. far from saturating the USB interface bandwidth. Some such drives become even slower when accessing data from several disparate areas of the drive concurrently. This is caused by limited RAM in the disk controller. The controller is unable to fit all the metadata needed to access the entirety of the drive in RAM at once, and will end up re-reading metadata from the flash medium as it changes read areas on the drive, often having to follow link chains to find it. Such metadata reads are often not parallelized by the drive even if the flash memory layout would allow it, and are often also implemented using dedicated code paths that are just slow.

The only sure way to deal with that is to do a bandwidth test. Suppose you have to verify archives a, b, c, d, e and f, all roughly in the same order of magnitude size-wise. They need to be sorted descending by size. First, time the verification of a, and compute the bandwidth BW_a = time_a / size_a. Then do the verification of b and c in parallel, and compute those bandwidths. As long as their sum sum_BW_bc is larger than BW_a, you're gaining performance. Then verify d, e and f in parallel and compute those bandwidths. As long as their sum sum_BW_def is larger than sum_BW_bc , you're gaining performance. And so on. Eventually you'll reach the number of streams where the total bandwidth drops compared to the previous test. The continue verifying the remaining archives using the previous, smaller number of streams.

This method can be applied to any drive, although the performance drop with mechanical drives is so sharp that it may unduly affect the overall performance of your verification process. Because of that, the parallel tests should be terminated when it's known that their bandwidth is less than 90% of the bandwidth of the preceding step. You can do that by recomputing the bandwidth every second the test runs, and terminating the test when the bandwidth is below the cutoff point.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .