1

I'm experimenting with using ZFS to deduplicate a large library of FLAC files. The purpose of this is twofold:

  1. Reduce storage utilization
  2. Reduce bandwidth needed to sync the library with cloud storage

Many of these files are of the same music tracks but from different physical media. This means that for the most part they are the same and usually close to the same size, which makes me think that they should benefit from block-level deduplication.

However in my testing I'm not seeing good results. When I create a pool and add three of these tracks (identical songs from different source media) zpool list reports 1.00 dedupe. If I copy all of the files (make exact duplicates of the three) dedupe climbs, so I know that it is enabled and functioning, but it's not finding any duplication in the original collection of files.

My first thought was that perhaps some of the variable header data (metadata tags, etc.) might be mis-aligning the bulk of the data in these files (the audio frames) but even making the header data consistent across the three files doesn't seem to have any impact on deduplication.

I'm considering taking alternate routes (testing other dedupe filesystems as well as some custom code) but since we're already using ZFS and I like the ZFS replication options, I'd prefer to use ZFS dedupe for this project; but perhaps it's simply not capable of working well with this sort of data.

Any feedback regarding tuning that might improve dedupe performance for this sort of dataset, or confirmation that ZFS dedupe is not the right tool for this job are appreciated.

1 Answer 1

4

I don't know about ZFS, but the FLAC codec has a lot of different parameters, and the structure inside a FLAC file is not byte-aligned in general. So any slight change in settings is likely to give a very different bytestream. Do you know if the "identical" files were flacced with the same software, the same software version, and the same settings, on the same architecture?

Assuming that there are some differences in the bytestream (which would explain your 1.0 result), a way to test this would be to decompress and recompress all the FLAC files on the same machine. (Of course this operation doesn't drop any data, as long as metadata is kept.)

7
  • +1: This is pretty much it. Just because it's the "same song" on 2 different albums doesn't mean it's the exact same mix, number of samples, amplitude, alignment, or anything else that would cause 2 encodes to be exactly the same. You're not, in general, going to get a significant level of dedupe out of a library of compressed files.
    – afrazier
    Oct 15, 2012 at 17:44
  • This is where I was going but I wanted to make sure I wasn't overlooking something (like recordsize, etc.) that might have an impact. I want a linefeed but it's not in the cards :) Oct 15, 2012 at 19:19
  • The files are definitely not identical (by design), just the same songs from the same albums. I thought perhaps with a large enough record size that similarities between the files might lead to some level of deduplication but perhaps that's just not the case. Oct 15, 2012 at 19:21
  • 1
    +1: Dan is correct. Given the nature of FLAC files with their myriad of encoding parameters and metadata, the chances of there being blocks of duplicate data across different files (even of the same song) is very low. You may also want to read up on ZFS dedup as many users have found it to not be the silver bullet they were hoping for. Oct 15, 2012 at 22:30
  • Agreed, what jasongullickson needs is something like Audio Dedupe (mindgems.com/products/Duplicate-MP3-Finder/…) for Linux. Oct 16, 2012 at 2:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .