Deduplication is the finding and removal of duplicate files.

learn more… | top users | synonyms (1)

0
votes
1answer
223 views

Find duplicates of a particular file

I downloaded an icon library from some website and I used several of the icons in a project. Now I need to find another icon from the library, but I can not find the library on my computer. Because I ...
1
vote
1answer
189 views

Check for duplicate files in two collections, ignoring duplicates within A

I have four similar large collections of website files. They are archived copies of a similar directory tree from different times in the past. I want to merge them all to the most recent collection, ...
1
vote
1answer
755 views

Detecting image duplicates in 7-Zip archives on Windows?

I have a large image gallery, and a few years later I realized that I get more and more duplicates. :) Basically, my gallery consists of 7-Zip archives containing the image sets. I saw there are some ...
1
vote
2answers
156 views

Is there something like hardlinks that split into separate files on modifications?

Let's say files orig and copy are huge but identical files. In order to save storage space one could therefore either make copy a soft-link to orig or hard-link both files to the same data. orig must ...
1
vote
1answer
740 views

md5deep not working with duplicated files in different directories

I want to use md5deep to check that some copied directories are copied correctly. But seems md5deep doesn't use the filenames stored in the genearted .md5-files when matching! This actually means, ...
2
votes
1answer
703 views

Deleting music file duplicates but retaining iTunes metadata

Back when I first imported the majority of my iTunes library I had the copy files to iTunes Media folder when adding to library option checked, so basically my whole music library was duplicated on ...
0
votes
1answer
159 views

Find RARs with duplicate content

I need a utility to find RAR files that contain duplicate data (i.e. files within the RAR that hash the same, but could have different names). I can open the RARs and see the CRCs are the same, but I ...
1
vote
2answers
1k views

Multicolumn spreadsheet: extract unique values in Column 1, given same values in Columns 3 and 4?

Here's a problem: given a (large) spreadsheet with userIDs in Column 1, a time/date pair in Column 2, a course code in Column 3, and a course title in Column 4: I want to produce a report showing how ...
2
votes
2answers
478 views

duplicate file/folder finder application for windows 7

I got old archives of my old computers, Now I decided to merge all these folders and organize them, I got many backups so I need app that will find duplicate files and delete / move them, any ideas?
1
vote
1answer
529 views

How can I consolidate duplicate entries in iTunes without losing statistical data?

I recently combined two of my PCs in a Frankensteinian attempt to optimize my hardware configuration. One of these PCs was much older and had a different collection of music on it than my current one ...
3
votes
4answers
3k views

Is there a diff utility that allows you to exclude columns?

For example, I have a text file, each line is a long string. I want to exclude 2 "segments" of this string, say columns 1-7 and 20-22. So the bottom 2 lines below would be a match: ...
4
votes
2answers
928 views

Software to detect mp3 almost-duplicates?

Because of some unfortunate circumstances I noticed I irrevertibly mixed up my sorted and retagged mp3s with an old backup. That means now I have files that are basically duplicates except for the id3 ...
1
vote
3answers
1k views

Confirm that two filesystems are identical, ignoring special files

/media/A and /media/B should be identical, but I want to confirm before deleting one. Duplicate file finders don't work, because they'll find two copies of the same file within B, for instance. I ...
8
votes
4answers
2k views

Is there a compression or archiver program for Windows that also does deduplication?

I'm looking for an archiver program that can perform deduplication (dedupe) on the files being archived. Upon unpacking the archive, the software would put back any files it removed during the ...
1
vote
0answers
222 views

Duplicate Images / Tags - sorting and removing

I have folders full of camera images that I have copied and backed up between my laptop, computer and external hard drive(s). I want to organise and sort them, and remove duplicates, so I have used a ...
1
vote
2answers
966 views

Two PDF files, identical except for possibly different cover pages — how to remove duplicates?

I have a bunch of PDF files which are articles downloaded from various Journal websites. The problem is, some of these PDF files are duplicates of one another. However, if I run a basic duplicate file ...
1
vote
1answer
438 views

Why is de-dupe feature of ZFS not working with FreeBSD 8.2

I have FreeBSD 8.2 installed on AMD/64 with 3 GB RAM and 690 GB pooled HD under ZFS. I have two copies of an identical file (180 MB video) in same directory and yet df -h indicates that both are using ...
1
vote
2answers
635 views

Windows 7 file-based backup service [closed]

I'm looking for a good replacement for Lazy Mirror, since it doesn't support Windows 7 well. Pros: One of the things I really loved about Lazy Mirror is that it always maintains a "full" backup, but ...
0
votes
1answer
607 views

Looking for a HDD tool that can perform deduplication and data compression

Hi experts and pros! And a happy new year 2011 to everyone! I'm looking for a "private" tool that could deduplicate my external HDD. I'm sure I have tons of files double on my Disk drives, and I ...
1
vote
3answers
181 views

Find copies of folders? (Not files)

I have a dozen of folders that are duplicates. Within them are a few dozen folders that are duplicates so i have a few thousand copies of the same files and folders. Many of them are exactly the same ...
5
votes
4answers
261 views

Locate near-duplicate files in a folder

I have a folder full of files, with a number of duplicate files in it. Unfortunately, in a number of cases, one version is an updated version of another, so a straight byte-match doesn't locate the ...
1
vote
4answers
652 views

Tool for merging multiple OFX files into one?

OFX is an XML format for financial transaction history and many banks, including mine, provide the option to download your transaction data in this format. Unfortunately, they often force you to ...
4
votes
1answer
1k views

Remove duplicate emails (Outlook 2003)

Sometimes our POP server has a fit and it re-sends a bunch of e-mail messages again to a particular user and I have to clean up those duplicate messages that ended up in the local outlook store. Is ...
2
votes
3answers
435 views

Backup of 5TB server

I'm looking for a software that will help me backup a server with 5TB of data. My wishlist: Backup on 1-2TB SATA disks Should need little to no configuration (just the directories to backup and the ...
0
votes
0answers
191 views

How can I remove duplicate messages from my inbox? [duplicate]

Possible Duplicate: Removing duplicate emails in Thunderbird For some reason known only to Thunderbird and Hotmail they decided to re-download all my e-mails, twice. So I now have about 20k ...
8
votes
6answers
3k views

Ways to deduplicate files

I want to simply backup and archive the files on several machines. Unfortunately, the files have some large files that are the same file but stored differently on different machines. For instance, ...
0
votes
3answers
416 views

Deduplication of two sets of contacts

One of our users has two sets of contacts in his email: 7000 or so from and old mail system, and even more in his current Apple Address Book. What we're trying to do is get the two sets of contacts ...
9
votes
5answers
6k views

Free Duplicate mp3 finder [closed]

Sometime back I had used a duplicate file finder for mp3 by analyzing the content.Unfortunately it was not free and the shareware had a lot of limitations. Are there any freeware/OSS ones to detect ...
52
votes
15answers
17k views

Which duplicate files and folders finders exist for Windows? [closed]

I need a free duplicate file finder/remover app, with ability to find duplicate files/folders by name and/or by size and to remove one of duplicates.
13
votes
6answers
3k views

What is the best method to remove duplicate image files from your computer?

I have a lot of duplicate image files on my Windows computer, in different subfolders and with different file names. What Python script or freeware program would you recommend for removing the ...