2

I know that setting dedup=on causes high memory usage and that it's almost never worth it (see e.g. https://serverfault.com/a/569099/208661). However I did turn it on in order to see how it works and put 20 GB on a 1.4 TB pool with a cache. The memory consumption (difference of memory usage after boot with and without zfs) is approx. 8 GB higher than with btrfs. After setting dedup=on it was noticable higher than with btrfs, but not 8 GB. Now, I turned dedup off and rebooted, but the memory consumption is still high (I'd say the same).

Will it remain high, i.e. do I have to destroy and recreate the pool in order to avoid this memory consumption, will it take time until it does down, can I do something (some sort of re-duplication) in order to speed up the process?

  • What "memory consumption" are you referring to? – Andrew Henle Nov 1 '16 at 9:58
4

Assuming your problems stem from dedup, it will not help much, as only new data is unduplicated, old data remains deduplicated and the dedup table also remains, see http://list.zfsonlinux.org/pipermail/zfs-discuss/2013-March/007668.html :

This is a very commonly-asked question. You can technically set dedup=off on a pool, and newly-written data will not be deduped. The pool will continue to work perfectly. However, the data structures required for all the existing deduped data will not go away, and nor will any of their performance implications.

The bottom line is that if you are disabling dedup to fix slowness caused by dedup, it won't work. In that case, you will need to send the data to another storage device, destroy your pool, create a new pool with dedup off, and then restore the data. There's currently no other way.

This would be quite quick in your case, as you only have 20GB to move.

Other good resources for the details behind dedup are:

| improve this answer | |
  • Why we have to destroy the pool? Isn't this enough if eventually all the files disappear that had dedup flag set to on? I have rolling backup and in few weeks it has replaced all the files in the zfs pool. Would this volume still have some "ill effects" from the previously enabled and then disabled dedup feature? – Janno Jun 22 at 0:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.