15

I'm trying to compress a 100GB with a good ratio, so I decided to use xz. I don't need it to be very fast, but since I have an 8-threads CPU, I was wondering how to take advantage of them.

I found there is a parallel implementation of xz: pixz. I found too that xz has a multithread option -T. I'm experimenting with both of them but I can't see any difference in their behavior.

What's the difference between them? Why are there 2 parallel implementations for xz?

Edit: I found there is another one: pxz.

Improve this question
2
  • 1
    What kind of difference did you check? Compressed file size or compression time? I think, pixz was started before xz implemented parallel variant. PS: try lrzip by kolivas, it can find duplicated parts of file and compress better.
    – osgx
    Mar 14, 2015 at 6:38
  • I was wondering about both size and time. Maybe each one was optimized for different circumstances like availability of free memory, number of processors and so... Thanks for the lrzip hint, I didn't know that one.
    – Dertalai
    Mar 15, 2015 at 18:53

1 Answer 1

Reset to default
38

I'm the author of pixz. As speculated above, I did write pixz before threading support was added to xz. I like to think that pixz prodded Lasse Collin to add multi-threading support to xz sooner.

There are still some differences between pixz and xz, of course. In xz's favour:

  • It has far more options
  • It's been tested by many more people
  • It probably has better code quality—pixz is a prototype gone rampant ;)

But there are still some reasons I like to use pixz:

  • It's much smaller, and easier to comprehend
  • It supports multi-threaded decompression, which xz does not
  • It has special behaviour on tarballs: While performing compression, it also creates an index of all the files in the tarball. This allows a single file to be extracted quickly, even if the tarball is huge.
Improve this answer
4
  • 6
    Never heard of pixz before, but intending to use it now just for that last point alone
    – Mark K Cowan
    Oct 20, 2016 at 17:10
  • Is pixz format with index enabled compatible with classic xz tool? How input file is partitioned when compressed in parallel?
    – osgx
    Mar 22, 2017 at 23:39
  • 4
    The pixz format is compatible with xz. If pixz adds an index, it will slightly change the output after decompression with xz—but the only change will be after the tar End-Of-Archive marker, and therefore ignored by tar. If you don't want an index, you can disable it with the '-t' option.
    – vasi
    Mar 23, 2017 at 15:20
  • 2
    Partitioning the input file into blocks is based on: * The xz dictionary size, which depends on the compression level. For the default level of '-6', it's 8 MiB. See the xz manpage for more details. * A multiplier, provided with the '-f' option. The default is 2.0. The default block size will therefore be 16 MiB. If you want more details about pixz, please check out the docs
    – vasi
    Mar 23, 2017 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.