0

I have a PIXZ-compressed (level -9) tarball containing around 4000 files (ordered, like pages in a book): compressed size is ~670M. Currently I am programmatically accessing these files using the standard way, i.e.

pixz -x <compressed_file_name> < tarball.tpxz | tar x -O

Based on metrics using time, extraction of the file takes on average 1.7 seconds. Since this is part of a programmatic process, I wanted to get that time down if possible, so I thought of splitting the tpxz archive into three smaller ~200M segments (each containing ~1000 files), with the expectation that pixz -x would run significantly faster against any one of these three segments than it does against the ~600M original. (I can predict which of the three segments contains a file required for the process.)

However to my surprise, the time metrics against the 200M segments are identical to those for the original: lookup/decompression still runs 1.7 seconds on average. As this is at odds both with intuition and with results in an extremal case - lookup/decompression against a -9-compressed tarball containing a single file completes in a trivial amount of time - I am curious why my segmentation strategy failed, and whether there are any other strategies folks can recommend to improve the performance of pixz lookups against largish files: 1.7 seconds is certainly acceptable, particularly given what you save in storage costs, but a faster time would be nice.

If there's some tarball size and/or archive file number threshold past which completion time remains roughly constant for pixz lookup/decompression jobs, it would be both interesting and useful to know that, so thanks in advance for any advice.

2
  • Of course the times are the same. Decompression is mainly a CPU job and that is the same regardless whether the archive has 3 entries or 3000 entries. There may be just a very small difference in disk time if you have an old fashioned spinning disk.
    – whs
    Jan 25, 2016 at 1:01
  • Then why the large difference in the extremal case? Or in a subsequent test with 40-file segments, lookup times decrease to .5 second Jan 25, 2016 at 1:14

1 Answer 1

0

I'm the author of pixz. Pixz already uses internal blocking for looking up files, that's why using "pixz -x file < my.tpxz" tends to be faster than just "tar -xJf my.tpxz file".

When you're using high compression, you block sizes end up huge, which makes extraction of a single file slower than lower compression settings. You can use "xz -lv" on your .tpxz file to see what sizes the blocks are.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .