I am working with some very large packet capture files (~150GB across 30 files) but because of how highly redundant packet captures are, these compress to ~7% (14.29x) of their originally file size using .zip files.

I was wondering if I could work with these files "natively" by creating a ZFS partation with the "compression=gzip-9" setting. I created the partition, copied the packet captures over, and the end result was a compression ratio of 20.83% (4.80x).

I am wondering why there is such a large discrepancy.


This is because each record (file block) is compressed independently (to allow reading from the middle of the file for large files). Set large record size:

zfs set recordsize=128k zpool/fsname

or create new filesystem with large record size:

zfs create -o recordsize=128k zpool/fsname

This setting affects only new files created in the filesystem fsname.

|improve this answer|||||
  • The stats I posted were with a 128k recordsize, however such a small block size would definitely be the root cause. – Chuu Mar 8 '13 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.