Deduplication is the finding and removal of duplicate files.

learn more… | top users | synonyms (1)

0
votes
1answer
532 views

md5deep not working with duplicated files in different directories

I want to use md5deep to check that some copied directories are copied correctly. But seems md5deep doesn't use the filenames stored in the genearted .md5-files when matching! This actually means, ...
2
votes
1answer
576 views

Deleting music file duplicates but retaining iTunes metadata

Back when I first imported the majority of my iTunes library I had the copy files to iTunes Media folder when adding to library option checked, so basically my whole music library was duplicated on ...
0
votes
1answer
153 views

Find RARs with duplicate content

I need a utility to find RAR files that contain duplicate data (i.e. files within the RAR that hash the same, but could have different names). I can open the RARs and see the CRCs are the same, but I ...
2
votes
2answers
397 views

Opensource duplicate finder for directories?

Is there an open source tool that will tell me if two directories have byte-for-byte identical content? I need a tool that confirms if two directories are the same. A listing of files that I need to ...
1
vote
2answers
1k views

Multicolumn spreadsheet: extract unique values in Column 1, given same values in Columns 3 and 4?

Here's a problem: given a (large) spreadsheet with userIDs in Column 1, a time/date pair in Column 2, a course code in Column 3, and a course title in Column 4: I want to produce a report showing how ...
1
vote
1answer
511 views

How can I consolidate duplicate entries in iTunes without losing statistical data?

I recently combined two of my PCs in a Frankensteinian attempt to optimize my hardware configuration. One of these PCs was much older and had a different collection of music on it than my current one ...
3
votes
4answers
2k views

Is there a diff utility that allows you to exclude columns?

For example, I have a text file, each line is a long string. I want to exclude 2 "segments" of this string, say columns 1-7 and 20-22. So the bottom 2 lines below would be a match: ...
4
votes
2answers
706 views

Software to detect mp3 almost-duplicates?

Because of some unfortunate circumstances I noticed I irrevertibly mixed up my sorted and retagged mp3s with an old backup. That means now I have files that are basically duplicates except for the id3 ...
1
vote
3answers
839 views

Confirm that two filesystems are identical, ignoring special files

/media/A and /media/B should be identical, but I want to confirm before deleting one. Duplicate file finders don't work, because they'll find two copies of the same file within B, for instance. I ...
7
votes
4answers
1k views

Is there a compression or archiver program for Windows that also does deduplication?

I'm looking for an archiver program that can perform deduplication (dedupe) on the files being archived. Upon unpacking the archive, the software would put back any files it removed during the ...
1
vote
0answers
209 views

Duplicate Images / Tags - sorting and removing

I have folders full of camera images that I have copied and backed up between my laptop, computer and external hard drive(s). I want to organise and sort them, and remove duplicates, so I have used a ...
1
vote
1answer
685 views

Two PDF files, identical except for possibly different cover pages — how to remove duplicates?

I have a bunch of PDF files which are articles downloaded from various Journal websites. The problem is, some of these PDF files are duplicates of one another. However, if I run a basic duplicate file ...
1
vote
1answer
437 views

Why is de-dupe feature of ZFS not working with FreeBSD 8.2

I have FreeBSD 8.2 installed on AMD/64 with 3 GB RAM and 690 GB pooled HD under ZFS. I have two copies of an identical file (180 MB video) in same directory and yet df -h indicates that both are using ...
1
vote
2answers
593 views

Windows 7 file-based backup service

I'm looking for a good replacement for Lazy Mirror, since it doesn't support Windows 7 well. Pros: One of the things I really loved about Lazy Mirror is that it always maintains a "full" backup, but ...
1
vote
1answer
500 views

Looking for a HDD tool that can perform deduplication and data compression

Hi experts and pros! And a happy new year 2011 to everyone! I'm looking for a "private" tool that could deduplicate my external HDD. I'm sure I have tons of files double on my Disk drives, and I ...
1
vote
3answers
163 views

Find copies of folders? (Not files)

I have a dozen of folders that are duplicates. Within them are a few dozen folders that are duplicates so i have a few thousand copies of the same files and folders. Many of them are exactly the same ...
5
votes
4answers
237 views

Locate near-duplicate files in a folder

I have a folder full of files, with a number of duplicate files in it. Unfortunately, in a number of cases, one version is an updated version of another, so a straight byte-match doesn't locate the ...
1
vote
4answers
510 views

Tool for merging multiple OFX files into one?

OFX is an XML format for financial transaction history and many banks, including mine, provide the option to download your transaction data in this format. Unfortunately, they often force you to ...
4
votes
1answer
1k views

Remove duplicate emails (Outlook 2003)

Sometimes our POP server has a fit and it re-sends a bunch of e-mail messages again to a particular user and I have to clean up those duplicate messages that ended up in the local outlook store. Is ...
2
votes
3answers
395 views

Backup of 5TB server

I'm looking for a software that will help me backup a server with 5TB of data. My wishlist: Backup on 1-2TB SATA disks Should need little to no configuration (just the directories to backup and the ...
0
votes
0answers
191 views

How can I remove duplicate messages from my inbox? [duplicate]

Possible Duplicate: Removing duplicate emails in Thunderbird For some reason known only to Thunderbird and Hotmail they decided to re-download all my e-mails, twice. So I now have about 20k ...
8
votes
6answers
3k views

Ways to deduplicate files

I want to simply backup and archive the files on several machines. Unfortunately, the files have some large files that are the same file but stored differently on different machines. For instance, ...
0
votes
3answers
398 views

Deduplication of two sets of contacts

One of our users has two sets of contacts in his email: 7000 or so from and old mail system, and even more in his current Apple Address Book. What we're trying to do is get the two sets of contacts ...
9
votes
5answers
5k views

Free Duplicate mp3 finder

Sometime back I had used a duplicate file finder for mp3 by analyzing the content.Unfortunately it was not free and the shareware had a lot of limitations. Are there any freeware/OSS ones to detect ...
48
votes
15answers
12k views

Which duplicate files and folders finders exist for Windows? [closed]

I need a free duplicate file finder/remover app, with ability to find duplicate files/folders by name and/or by size and to remove one of duplicates.