Deduplication is the finding and removal of duplicate files.

learn more… | top users | synonyms (1)

51
votes
15answers
15k views

Which duplicate files and folders finders exist for Windows? [closed]

I need a free duplicate file finder/remover app, with ability to find duplicate files/folders by name and/or by size and to remove one of duplicates.
16
votes
3answers
366 views

How to deduplicate 40TB of data?

I've inherited a research cluster with ~40TB of data across three filesystems. The data stretches back almost 15 years, and there are most likely a good amount of duplicates as researchers copy each ...
9
votes
5answers
5k views

Free Duplicate mp3 finder

Sometime back I had used a duplicate file finder for mp3 by analyzing the content.Unfortunately it was not free and the shareware had a lot of limitations. Are there any freeware/OSS ones to detect ...
8
votes
4answers
2k views

How can I have two files with the same name in a directory when mounted with NFS?

I have a C++ application test that creates 10,000 files in an NFS mounted directory, but my test recently failed once due to one file appearing twice with the same name in that directory with all the ...
8
votes
6answers
3k views

Ways to deduplicate files

I want to simply backup and archive the files on several machines. Unfortunately, the files have some large files that are the same file but stored differently on different machines. For instance, ...
7
votes
4answers
2k views

Is there a compression or archiver program for Windows that also does deduplication?

I'm looking for an archiver program that can perform deduplication (dedupe) on the files being archived. Upon unpacking the archive, the software would put back any files it removed during the ...
6
votes
3answers
2k views

Why do I have identical files in one directory, in Windows 7?

I just rebuilt my system after a new Power supply fired my CPU, MBD, Video card & a Blu-Ray drive. During all this I had to restore Windows from a backup (sector copy). All seems well, but ...
5
votes
4answers
250 views

Locate near-duplicate files in a folder

I have a folder full of files, with a number of duplicate files in it. Unfortunately, in a number of cases, one version is an updated version of another, so a straight byte-match doesn't locate the ...
4
votes
2answers
812 views

Software to detect mp3 almost-duplicates?

Because of some unfortunate circumstances I noticed I irrevertibly mixed up my sorted and retagged mp3s with an old backup. That means now I have files that are basically duplicates except for the id3 ...
4
votes
3answers
2k views

iTunes - One track in several albums

I'm using Windows 7 to do this. I have quite an extensive library of music and now I have quite a lot of duplicate tracks but in different albums. Obviously I don't really want duplicate files but I'd ...
4
votes
1answer
702 views

How to find duplicate images and compare their meta-data?

Is there an application for Windows that finds duplicate images and lets quickly preview the image and, what is most important, the meta-data of the image files. I can see there are programs for ...
4
votes
1answer
1k views

Remove duplicate emails (Outlook 2003)

Sometimes our POP server has a fit and it re-sends a bunch of e-mail messages again to a particular user and I have to clean up those duplicate messages that ended up in the local outlook store. Is ...
3
votes
4answers
2k views

Is there a diff utility that allows you to exclude columns?

For example, I have a text file, each line is a long string. I want to exclude 2 "segments" of this string, say columns 1-7 and 20-22. So the bottom 2 lines below would be a match: ...
3
votes
3answers
68 views

Is there such a thing as a file hosted container which deduplicates data held within?

Background I have backups of a website which stores all of it's data into a single file. This file is several gigs large and I have many different backups of this file. Most of the data within is ...
3
votes
1answer
59 views

Ensure the download is not a dupe

I want to (recursively) download a directory of files to a location from a WebDav Server. If the file is alreay present (somewhere there), it shall not be downloaded again. However the folder ...
3
votes
3answers
441 views

Fuzzy Duplicate File Finder?

Is there any GUI application (either for Windows or Linux) that implements fuzzy hashing algorithms like ssdeep or sdhash to find near duplicate files (mostly text files)?
3
votes
1answer
113 views

Painless way of consolidating files across multiple machines/OSes

Just bought a NAS. So I thought I'd get all our photos, media files and pdfs consolidated, de-duplicated, de-junked and virus-checked and stick them all on it. We have 3 laptops, one running Windows, ...
3
votes
0answers
254 views

Rsync, how can I delete remote duplicates?

I'm trying to delete remote duplicate files without transferring the non-existing files, and without updating the existing files. If I specify both --existing and --ignore-existing (along with -av ...
2
votes
2answers
773 views

Remove duplicate lines from a very large text-file [duplicate]

I have a very large text-file (> 50 GB), but most lines are duplicate, so I want to remove them. Is there any way to remove duplicates lines from a file, and handle files > 2GB? Because every method I ...
2
votes
1answer
624 views

ZFS compressed and deduped filesystem on file

I can create squashed filesystems with mksquashfs in this way: mksquashfs dir-1 dir-2 ... dir-n filesystem-on-file.squash -comp xz -Xbcj x86 -Xdict-size 1048576 -b 1024k mksquashfs provides ...
2
votes
2answers
316 views

Duplicate File Scanner

I have a 15TB storage network, and I am down to about 2.5TB now (due to a large amount of duplicates). I have tried many scanners, but I have had little success, eventually they all crash due to the ...
2
votes
1answer
308 views

How to de-dupe identical photos that have a slightly different file size?

I imported many photos using the new "camera import" feature of Dropbox. Many of those were duplicates of photos previously imported by direct copying from the camera. Strangely, the Dropbox import ...
2
votes
2answers
463 views

duplicate file/folder finder application for windows 7

I got old archives of my old computers, Now I decided to merge all these folders and organize them, I got many backups so I need app that will find duplicate files and delete / move them, any ideas?
2
votes
2answers
488 views

Deduplicate files using the command line in OSX Lion

I am using OSX 10.8.4 and would like to remove duplicate files in a single flat directory if an md5 hash matches. I have looked up several answers but none of them work for me, I think because of ...
2
votes
1answer
645 views

Deleting music file duplicates but retaining iTunes metadata

Back when I first imported the majority of my iTunes library I had the copy files to iTunes Media folder when adding to library option checked, so basically my whole music library was duplicated on ...
2
votes
3answers
417 views

Backup of 5TB server

I'm looking for a software that will help me backup a server with 5TB of data. My wishlist: Backup on 1-2TB SATA disks Should need little to no configuration (just the directories to backup and the ...
2
votes
2answers
330 views

Deduplicate Files inside multiple .tar.gz archives

I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB). I would like to archive these files and deduplicate (solid compression) files ...
1
vote
2answers
149 views

Is there something like hardlinks that split into separate files on modifications?

Let's say files orig and copy are huge but identical files. In order to save storage space one could therefore either make copy a soft-link to orig or hard-link both files to the same data. orig must ...
1
vote
1answer
4k views

View ZFS deduplication ratio on a dataset

I have a tank consisting of several datasets, only one of which is configured to use deduplication. How can I see the ratio for this dataset? I get a ratio of 1.00x for the whole pool but I imagine ...
1
vote
3answers
447 views

Program for remove exact duplicate files while caching search results

We need a Windows 7 program to remove/check the duplicates but our situation is somewhat different than the standard one for which there are enough programs. We have a fairly large static archive ...
1
vote
1answer
683 views

Detecting image duplicates in 7-Zip archives on Windows?

I have a large image gallery, and a few years later I realized that I get more and more duplicates. :) Basically, my gallery consists of 7-Zip archives containing the image sets. I saw there are some ...
1
vote
1answer
626 views

md5deep not working with duplicated files in different directories

I want to use md5deep to check that some copied directories are copied correctly. But seems md5deep doesn't use the filenames stored in the genearted .md5-files when matching! This actually means, ...
1
vote
2answers
1k views

Multicolumn spreadsheet: extract unique values in Column 1, given same values in Columns 3 and 4?

Here's a problem: given a (large) spreadsheet with userIDs in Column 1, a time/date pair in Column 2, a course code in Column 3, and a course title in Column 4: I want to produce a report showing how ...
1
vote
3answers
1k views

Confirm that two filesystems are identical, ignoring special files

/media/A and /media/B should be identical, but I want to confirm before deleting one. Duplicate file finders don't work, because they'll find two copies of the same file within B, for instance. I ...
1
vote
1answer
437 views

Why is de-dupe feature of ZFS not working with FreeBSD 8.2

I have FreeBSD 8.2 installed on AMD/64 with 3 GB RAM and 690 GB pooled HD under ZFS. I have two copies of an identical file (180 MB video) in same directory and yet df -h indicates that both are using ...
1
vote
1answer
71 views

Deleting all but latest revision of a file

So I have these massive lists of drawings that I've done for work and I'd like to be able to dump them all into one folder and run a batch that would delete all of the older revs and leave the highest ...
1
vote
1answer
222 views

Is there a way to remove duplicate emails from a remote account?

I have a user who has multiple duplicated emails across multiple folders on his IMAP account. How he managed to create them is beyond me, but mine is not to reason why, just to fix it! Can anyone ...
1
vote
1answer
529 views

How can I consolidate duplicate entries in iTunes without losing statistical data?

I recently combined two of my PCs in a Frankensteinian attempt to optimize my hardware configuration. One of these PCs was much older and had a different collection of music on it than my current one ...
1
vote
2answers
834 views

Two PDF files, identical except for possibly different cover pages — how to remove duplicates?

I have a bunch of PDF files which are articles downloaded from various Journal websites. The problem is, some of these PDF files are duplicates of one another. However, if I run a basic duplicate file ...
1
vote
1answer
545 views

Looking for a HDD tool that can perform deduplication and data compression

Hi experts and pros! And a happy new year 2011 to everyone! I'm looking for a "private" tool that could deduplicate my external HDD. I'm sure I have tons of files double on my Disk drives, and I ...
1
vote
3answers
177 views

Find copies of folders? (Not files)

I have a dozen of folders that are duplicates. Within them are a few dozen folders that are duplicates so i have a few thousand copies of the same files and folders. Many of them are exactly the same ...
1
vote
4answers
559 views

Tool for merging multiple OFX files into one?

OFX is an XML format for financial transaction history and many banks, including mine, provide the option to download your transaction data in this format. Unfortunately, they often force you to ...
1
vote
1answer
36 views

In Windows, is there any software to prevent duplication of files in a folder?

Let's say I have two files c:\f1\data.txt and c:\f2\data.txt. If I give the command copy c:\f2\data.txt c:\f1\data\ I will receive a warning about the potential owerwriting. I will receive the ...
1
vote
0answers
34 views

Which are good compression formats for an eternal version of a logfile? [closed]

I'm experimenting with logrotate and trying to set up an eternal version of a high volume logfile, e.g. /var/log/syslog, i.e. all entries should be preserved from the second the OS (Ubuntu 14.04) ...
1
vote
1answer
180 views

Safety of redirecting System folders (e.g. : program files ,users,… ) using symbolic links?

Can redirecting system folders as mentioned above using symbolic links cause problems to the system or some applications? And is there any problem if I redirect the folders to another partition on ...
1
vote
1answer
325 views

What is the best way to merge two sets of files, but for duplicates to select the older file in Windows?

I have stumbled into a dumb usage case. I want to merge two folders in a particular way (I guess these are called folders now, I still think of them as directories, but whatever). The folders have ...
1
vote
4answers
420 views

Duplicate name exists solution

I have about 70 pc's with exactly same hardware. I decided to automate turning on and off. I took 1 PC. Here is what I've done: Changed bios configuration so that now pc's waking when I turn on AC ...
1
vote
0answers
401 views

How to deduplicate files when storing in the SDFS file system using OpenDedup

I followed the link, http://www.opendedup.org/quickstart for installing OpenDedup in my linux machine (Ubuntu 12.04 - 64 bit). I created a SDFS volume and mounted it. I copied two similar Ubuntu VMs, ...
1
vote
1answer
187 views

Check for duplicate files in two collections, ignoring duplicates within A

I have four similar large collections of website files. They are archived copies of a similar directory tree from different times in the past. I want to merge them all to the most recent collection, ...
1
vote
0answers
216 views

Duplicate Images / Tags - sorting and removing

I have folders full of camera images that I have copied and backed up between my laptop, computer and external hard drive(s). I want to organise and sort them, and remove duplicates, so I have used a ...