Deduplication is the finding and removal of duplicate files.

learn more… | top users | synonyms (1)

49
votes
15answers
13k views

Which duplicate files and folders finders exist for Windows? [closed]

I need a free duplicate file finder/remover app, with ability to find duplicate files/folders by name and/or by size and to remove one of duplicates.
13
votes
1answer
230 views

How to deduplicate 40TB of data?

I've inherited a research cluster with ~40TB of data across three filesystems. The data stretches back almost 15 years, and there are most likely a good amount of duplicates as researchers copy each ...
9
votes
5answers
5k views

Free Duplicate mp3 finder

Sometime back I had used a duplicate file finder for mp3 by analyzing the content.Unfortunately it was not free and the shareware had a lot of limitations. Are there any freeware/OSS ones to detect ...
8
votes
4answers
1k views

How can I have two files with the same name in a directory when mounted with NFS?

I have a C++ application test that creates 10,000 files in an NFS mounted directory, but my test recently failed once due to one file appearing twice with the same name in that directory with all the ...
8
votes
6answers
3k views

Ways to deduplicate files

I want to simply backup and archive the files on several machines. Unfortunately, the files have some large files that are the same file but stored differently on different machines. For instance, ...
7
votes
4answers
2k views

Is there a compression or archiver program for Windows that also does deduplication?

I'm looking for an archiver program that can perform deduplication (dedupe) on the files being archived. Upon unpacking the archive, the software would put back any files it removed during the ...
6
votes
3answers
2k views

Why do I have identical files in one directory, in Windows 7?

I just rebuilt my system after a new Power supply fired my CPU, MBD, Video card & a Blu-Ray drive. During all this I had to restore Windows from a backup (sector copy). All seems well, but ...
5
votes
4answers
239 views

Locate near-duplicate files in a folder

I have a folder full of files, with a number of duplicate files in it. Unfortunately, in a number of cases, one version is an updated version of another, so a straight byte-match doesn't locate the ...
4
votes
2answers
740 views

Software to detect mp3 almost-duplicates?

Because of some unfortunate circumstances I noticed I irrevertibly mixed up my sorted and retagged mp3s with an old backup. That means now I have files that are basically duplicates except for the id3 ...
4
votes
1answer
584 views

How to find duplicate images and compare their meta-data?

Is there an application for Windows that finds duplicate images and lets quickly preview the image and, what is most important, the meta-data of the image files. I can see there are programs for ...
4
votes
1answer
1k views

Remove duplicate emails (Outlook 2003)

Sometimes our POP server has a fit and it re-sends a bunch of e-mail messages again to a particular user and I have to clean up those duplicate messages that ended up in the local outlook store. Is ...
3
votes
4answers
2k views

Is there a diff utility that allows you to exclude columns?

For example, I have a text file, each line is a long string. I want to exclude 2 "segments" of this string, say columns 1-7 and 20-22. So the bottom 2 lines below would be a match: ...
3
votes
1answer
59 views

Ensure the download is not a dupe

I want to (recursively) download a directory of files to a location from a WebDav Server. If the file is alreay present (somewhere there), it shall not be downloaded again. However the folder ...
3
votes
3answers
372 views

Fuzzy Duplicate File Finder?

Is there any GUI application (either for Windows or Linux) that implements fuzzy hashing algorithms like ssdeep or sdhash to find near duplicate files (mostly text files)?
3
votes
3answers
2k views

iTunes - One track in several albums

I'm using Windows 7 to do this. I have quite an extensive library of music and now I have quite a lot of duplicate tracks but in different albums. Obviously I don't really want duplicate files but I'd ...
3
votes
1answer
106 views

Painless way of consolidating files across multiple machines/OSes

Just bought a NAS. So I thought I'd get all our photos, media files and pdfs consolidated, de-duplicated, de-junked and virus-checked and stick them all on it. We have 3 laptops, one running Windows, ...
3
votes
0answers
234 views

Rsync, how can I delete remote duplicates?

I'm trying to delete remote duplicate files without transferring the non-existing files, and without updating the existing files. If I specify both --existing and --ignore-existing (along with -av ...
2
votes
2answers
400 views

Opensource duplicate finder for directories?

Is there an open source tool that will tell me if two directories have byte-for-byte identical content? I need a tool that confirms if two directories are the same. A listing of files that I need to ...
2
votes
2answers
601 views

Remove duplicate lines from a very large text-file [duplicate]

I have a very large text-file (> 50 GB), but most lines are duplicate, so I want to remove them. Is there any way to remove duplicates lines from a file, and handle files > 2GB? Because every method I ...
2
votes
3answers
63 views

Is there such a thing as a file hosted container which deduplicates data held within?

Background I have backups of a website which stores all of it's data into a single file. This file is several gigs large and I have many different backups of this file. Most of the data within is ...
2
votes
1answer
536 views

ZFS compressed and deduped filesystem on file

I can create squashed filesystems with mksquashfs in this way: mksquashfs dir-1 dir-2 ... dir-n filesystem-on-file.squash -comp xz -Xbcj x86 -Xdict-size 1048576 -b 1024k mksquashfs provides ...
2
votes
2answers
305 views

Duplicate File Scanner

I have a 15TB storage network, and I am down to about 2.5TB now (due to a large amount of duplicates). I have tried many scanners, but I have had little success, eventually they all crash due to the ...
2
votes
1answer
296 views

How to de-dupe identical photos that have a slightly different file size?

I imported many photos using the new "camera import" feature of Dropbox. Many of those were duplicates of photos previously imported by direct copying from the camera. Strangely, the Dropbox import ...
2
votes
1answer
597 views

Deleting music file duplicates but retaining iTunes metadata

Back when I first imported the majority of my iTunes library I had the copy files to iTunes Media folder when adding to library option checked, so basically my whole music library was duplicated on ...
2
votes
3answers
401 views

Backup of 5TB server

I'm looking for a software that will help me backup a server with 5TB of data. My wishlist: Backup on 1-2TB SATA disks Should need little to no configuration (just the directories to backup and the ...
2
votes
1answer
387 views

Deduplicate files using the command line in OSX Lion

I am using OSX 10.8.4 and would like to remove duplicate files in a single flat directory if an md5 hash matches. I have looked up several answers but none of them work for me, I think because of ...
2
votes
1answer
281 views

Deduplicate Files inside multiple .tar.gz archives

I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB). I would like to archive these files and deduplicate (solid compression) files ...
1
vote
2answers
137 views

Is there something like hardlinks that split into separate files on modifications?

Let's say files orig and copy are huge but identical files. In order to save storage space one could therefore either make copy a soft-link to orig or hard-link both files to the same data. orig must ...
1
vote
1answer
3k views

View ZFS deduplication ratio on a dataset

I have a tank consisting of several datasets, only one of which is configured to use deduplication. How can I see the ratio for this dataset? I get a ratio of 1.00x for the whole pool but I imagine ...
1
vote
3answers
415 views

Program for remove exact duplicate files while caching search results

We need a Windows 7 program to remove/check the duplicates but our situation is somewhat different than the standard one for which there are enough programs. We have a fairly large static archive ...
1
vote
1answer
633 views

Detecting image duplicates in 7-Zip archives on Windows?

I have a large image gallery, and a few years later I realized that I get more and more duplicates. :) Basically, my gallery consists of 7-Zip archives containing the image sets. I saw there are some ...
1
vote
1answer
553 views

md5deep not working with duplicated files in different directories

I want to use md5deep to check that some copied directories are copied correctly. But seems md5deep doesn't use the filenames stored in the genearted .md5-files when matching! This actually means, ...
1
vote
2answers
1k views

Multicolumn spreadsheet: extract unique values in Column 1, given same values in Columns 3 and 4?

Here's a problem: given a (large) spreadsheet with userIDs in Column 1, a time/date pair in Column 2, a course code in Column 3, and a course title in Column 4: I want to produce a report showing how ...
1
vote
3answers
919 views

Confirm that two filesystems are identical, ignoring special files

/media/A and /media/B should be identical, but I want to confirm before deleting one. Duplicate file finders don't work, because they'll find two copies of the same file within B, for instance. I ...
1
vote
1answer
437 views

Why is de-dupe feature of ZFS not working with FreeBSD 8.2

I have FreeBSD 8.2 installed on AMD/64 with 3 GB RAM and 690 GB pooled HD under ZFS. I have two copies of an identical file (180 MB video) in same directory and yet df -h indicates that both are using ...
1
vote
1answer
46 views

Deleting all but latest revision of a file

So I have these massive lists of drawings that I've done for work and I'd like to be able to dump them all into one folder and run a batch that would delete all of the older revs and leave the highest ...
1
vote
1answer
217 views

Is there a way to remove duplicate emails from a remote account?

I have a user who has multiple duplicated emails across multiple folders on his IMAP account. How he managed to create them is beyond me, but mine is not to reason why, just to fix it! Can anyone ...
1
vote
1answer
514 views

How can I consolidate duplicate entries in iTunes without losing statistical data?

I recently combined two of my PCs in a Frankensteinian attempt to optimize my hardware configuration. One of these PCs was much older and had a different collection of music on it than my current one ...
1
vote
1answer
738 views

Two PDF files, identical except for possibly different cover pages — how to remove duplicates?

I have a bunch of PDF files which are articles downloaded from various Journal websites. The problem is, some of these PDF files are duplicates of one another. However, if I run a basic duplicate file ...
1
vote
1answer
513 views

Looking for a HDD tool that can perform deduplication and data compression

Hi experts and pros! And a happy new year 2011 to everyone! I'm looking for a "private" tool that could deduplicate my external HDD. I'm sure I have tons of files double on my Disk drives, and I ...
1
vote
3answers
170 views

Find copies of folders? (Not files)

I have a dozen of folders that are duplicates. Within them are a few dozen folders that are duplicates so i have a few thousand copies of the same files and folders. Many of them are exactly the same ...
1
vote
4answers
526 views

Tool for merging multiple OFX files into one?

OFX is an XML format for financial transaction history and many banks, including mine, provide the option to download your transaction data in this format. Unfortunately, they often force you to ...
1
vote
1answer
142 views

Safety of redirecting System folders (e.g. : program files ,users,… ) using symbolic links?

Can redirecting system folders as mentioned above using symbolic links cause problems to the system or some applications? And is there any problem if I redirect the folders to another partition on ...
1
vote
1answer
264 views

What is the best way to merge two sets of files, but for duplicates to select the older file in Windows?

I have stumbled into a dumb usage case. I want to merge two folders in a particular way (I guess these are called folders now, I still think of them as directories, but whatever). The folders have ...
1
vote
4answers
354 views

Duplicate name exists solution

I have about 70 pc's with exactly same hardware. I decided to automate turning on and off. I took 1 PC. Here is what I've done: Changed bios configuration so that now pc's waking when I turn on AC ...
1
vote
0answers
374 views

How to deduplicate files when storing in the SDFS file system using OpenDedup

I followed the link, http://www.opendedup.org/quickstart for installing OpenDedup in my linux machine (Ubuntu 12.04 - 64 bit). I created a SDFS volume and mounted it. I copied two similar Ubuntu VMs, ...
1
vote
1answer
134 views

File synchronising software

Throughout my PC I have to have exactly the same files in more than one location (for one reason or another) and it's getting really difficult to keep track of them all and keep them all up-to-date ...
1
vote
1answer
185 views

Check for duplicate files in two collections, ignoring duplicates within A

I have four similar large collections of website files. They are archived copies of a similar directory tree from different times in the past. I want to merge them all to the most recent collection, ...
1
vote
0answers
213 views

Duplicate Images / Tags - sorting and removing

I have folders full of camera images that I have copied and backed up between my laptop, computer and external hard drive(s). I want to organise and sort them, and remove duplicates, so I have used a ...
1
vote
2answers
602 views

Windows 7 file-based backup service

I'm looking for a good replacement for Lazy Mirror, since it doesn't support Windows 7 well. Pros: One of the things I really loved about Lazy Mirror is that it always maintains a "full" backup, but ...