Deduplication is the finding and removal of duplicate files.

learn more… | top users | synonyms (1)

0
votes
2answers
4k views

How to Enable Data Deduplication in Windows 8.1 client

How can i enable Data Deduplication in Windows 8.1 x64 client. http://weikingteh.wordpress.com/2013/01/15/how-to-enable-data-deduplication-in-windows-8/ The above link discusses the same for a ...
5
votes
3answers
292 views

Locate near-duplicate files in a folder

I have a folder full of files, with a number of duplicate files in it. Unfortunately, in a number of cases, one version is an updated version of another, so a straight byte-match doesn't locate the ...
2
votes
2answers
1k views

Remove duplicate lines from a very large text-file [duplicate]

I have a very large text-file (> 50 GB), but most lines are duplicate, so I want to remove them. Is there any way to remove duplicates lines from a file, and handle files > 2GB? Because every method I ...
0
votes
1answer
39 views

Excel: Copy all cell values from an entire sheet to a new sheet, sort and deduplicate into a single list of unique values

The data (all text values) I have is scattered across the sheet (multiple rows and columns with gaps inbetween) and it is all calculated using formulas now I would like to make sure that I know what ...
5
votes
4answers
31k views

How to remove duplicate row based on some columns

I have a excel sheet which contains duplicate I want to remove a row if its A C D E F columns are same (Ignore B while calculating duplicates but remove it while removing a row) At the moment it ...
0
votes
1answer
35 views

How to delete one of each set of duplicate values in Excel?

I have a spreadsheet that contains many duplicates. Instead of removing all of the duplicates, I would like to remove one of each duplicate. I want to remove one record from each set of duplicates. ...
0
votes
1answer
56 views

Find duplicate media files and auto organize, on Linux

Is there an application similar to MediaMoney (on Windows), but on Linux (Debian or Ubuntu) that given when a disk-drive or a root directory, can search across it thoroughly, identifying duplicate ...
1
vote
0answers
69 views

using btrfs features to rapidly and efficiently combine large files

My aim is to use btrfs features to combine several 3GB files into a single 9GB file without producing substantial i/o nor using additional disk space. If this works for this set of data, I plan to ...
0
votes
1answer
45 views

Remove Duplicates inconsistently choosing which records to delete

So generally when using the remove duplicates function in Excel the oldest record will be kept - and that's exactly what I want to happen! But recently when running Remove Duplicates, it is ...
7
votes
3answers
3k views

Why do I have identical files in one directory, in Windows 7?

I just rebuilt my system after a new Power supply fired my CPU, MBD, Video card & a Blu-Ray drive. During all this I had to restore Windows from a backup (sector copy). All seems well, but ...
1
vote
2answers
565 views

Program for remove exact duplicate files while caching search results

We need a Windows 7 program to remove/check the duplicates but our situation is somewhat different than the standard one for which there are enough programs. We have a fairly large static archive ...
2
votes
3answers
530 views

Deduplicate Files inside multiple .tar.gz archives

I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB). I would like to archive these files and deduplicate (solid compression) files ...
1
vote
3answers
200 views

Find copies of folders? (Not files)

I have a dozen of folders that are duplicates. Within them are a few dozen folders that are duplicates so i have a few thousand copies of the same files and folders. Many of them are exactly the same ...
0
votes
4answers
429 views

Deduplication of two sets of contacts

One of our users has two sets of contacts in his email: 7000 or so from and old mail system, and even more in his current Apple Address Book. What we're trying to do is get the two sets of contacts ...
1
vote
0answers
42 views

Using btrfs cow functionallity to rapidly and efficiently subset large files

[My aim is to use btrfs cow features to subset 30GB to 300GB files into several 10GB files without producing substantial i/o nor using additional disk space. i.e.: 300GB dna_original.fastq 10GB ...
4
votes
2answers
612 views

How to replace all duplicate files with hard links?

I have two folders containing various files. Some of the files from the first folder have an exact copy in the second folder. I would like to replace those with a hard link. How can I do that?
2
votes
1answer
239 views

Duplicate Images / Tags - sorting and removing

I have folders full of camera images that I have copied and backed up between my laptop, computer and external hard drive(s). I want to organise and sort them, and remove duplicates, so I have used a ...
1
vote
2answers
261 views

Removing files with same name and different extension

I have a large music collection. It is organized in the following structure: artist / album / trackTitle. Over the course of time, in some directories, I have duplicate-ish files; most of my files ...
0
votes
2answers
61 views

In Windows, is there any software to prevent duplication of files in a folder? [closed]

Let's say I have two files c:\f1\data.txt and c:\f2\data.txt. If I issue the command copy c:\f2\data.txt c:\f1\data\ I will receive a warning about the potential overwriting. I will receive the ...
0
votes
2answers
130 views

Undo Creating Symbolic Links by copying back the original, rename the file and deleting the link

I used a windows 7 utility to delete exact byte per byte duplicate files and replace them with a symbolic link before I knew that the symbolic link type is not allowed in our corporate domain. I need ...
13
votes
6answers
3k views

What is the best method to remove duplicate image files from your computer?

I have a lot of duplicate image files on my Windows computer, in different subfolders and with different file names. What Python script or freeware program would you recommend for removing the ...
0
votes
0answers
24 views

Receive only 1 message when I'm listed as the recipient in 2 ways (Outlook 2013)

As a result of the following scenario, two identical emails appear in my Outlook 2013 inbox: to: my_email@my_work_domain.com cc: email_distribution_list@my_work_domain.com (which my email is ...
0
votes
0answers
43 views

Is in-line data de-duplication worthwhile when backing up for a photographer?

I create RAW image files plus a small selection of JPEG files derived from the RAW masters. The JPEGs once created are very, very rarely edited again but when they are because they are recompressed, ...
2
votes
1answer
27 views

How do I use an image multiple times in a OneNote document while storing the image only once?

I'm using OneNote 2013 for family history. I have created a page for each person with photos of and documents about that person. In several cases I have group photos with many family members in ...
0
votes
1answer
2k views

WMP 12.0 adds duplicate entries for same file mapped network drive

Windows Media Player 12 is adding duplicate entries for each file on my mapped network drive. When I try and rebuild the index by going to %UserProfile%\Appdata\Local\Microsoft\Windows Media, I am ...
0
votes
0answers
102 views

Copy only files that don't exist regardless of file name?

My dropbox account apparently renames all auto uploaded images. I'm about to clear my phone and I would like to verify that all images are in the dropbox folder before deleting them. If they had their ...
1
vote
1answer
2k views

Efficient removal of duplicate records across multiple Excel sheets

I have a workbook with 2 sheets. Sheet 1 has 3 columns (1A, 1B and 1C) and sheet 2 has 2 columns (2A and 2B). Now I need to delete rows from sheet 1 if 1A and 1B match any row in sheet 2 (i.e. 1A=2A ...
9
votes
5answers
6k views

Free Duplicate mp3 finder [closed]

Sometime back I had used a duplicate file finder for mp3 by analyzing the content.Unfortunately it was not free and the shareware had a lot of limitations. Are there any freeware/OSS ones to detect ...
0
votes
1answer
319 views

ZFS + dedup: copy many small files fast

I'm running a ZFS backup server (with deduplication) to backup websites and store them with retention. To do this, I transfer all files to the backup server during the first backup, and make a copy of ...
1
vote
2answers
1k views

Two PDF files, identical except for possibly different cover pages — how to remove duplicates?

I have a bunch of PDF files which are articles downloaded from various Journal websites. The problem is, some of these PDF files are duplicates of one another. However, if I run a basic duplicate file ...
0
votes
1answer
49 views

Is it possible to perform Deduplication recursively?

If anybody is familiar with the concept of Deduplication (If not,read link on wikipedia) http://en.wikipedia.org/wiki/Data_deduplication Is it possible to perform Deduplication recursively i.e. ...
0
votes
2answers
73 views

Virtual folder combining a read-only and read-write directory (file deduplication)

Is it possible in linux to have a directory that contains all the contents of another directory (in a read-only fashion) but you are able to write to the directory without it affecting the 'parent' ...
53
votes
15answers
20k views

Which duplicate files and folders finders exist for Windows? [closed]

I need a free duplicate file finder/remover app, with ability to find duplicate files/folders by name and/or by size and to remove one of duplicates.
17
votes
3answers
700 views

How to deduplicate 40TB of data?

I've inherited a research cluster with ~40TB of data across three filesystems. The data stretches back almost 15 years, and there are most likely a good amount of duplicates as researchers copy each ...
2
votes
2answers
990 views

Deduplicate files using the command line in OSX Lion

I am using OSX 10.8.4 and would like to remove duplicate files in a single flat directory if an md5 hash matches. I have looked up several answers but none of them work for me, I think because of ...
3
votes
1answer
120 views

Painless way of consolidating files across multiple machines/OSes

Just bought a NAS. So I thought I'd get all our photos, media files and pdfs consolidated, de-duplicated, de-junked and virus-checked and stick them all on it. We have 3 laptops, one running Windows, ...
2
votes
2answers
487 views

duplicate file/folder finder application for windows 7

I got old archives of my old computers, Now I decided to merge all these folders and organize them, I got many backups so I need app that will find duplicate files and delete / move them, any ideas?
0
votes
2answers
82 views

Duplicate Deletion Script does nothing

An associate of mine wrote a script for me that should theoretically delete files that have the same name, regardless of their file type. Specifically, it checks mp3, m4a, wma, wav files. However, ...
1
vote
1answer
243 views

Deleting all but latest revision of a file

So I have these massive lists of drawings that I've done for work and I'd like to be able to dump them all into one folder and run a batch that would delete all of the older revs and leave the highest ...
1
vote
1answer
558 views

What is the best way to merge two sets of files, but for duplicates to select the older file in Windows?

I have stumbled into a dumb usage case. I want to merge two folders in a particular way (I guess these are called folders now, I still think of them as directories, but whatever). The folders have ...
3
votes
3answers
83 views

Is there such a thing as a file hosted container which deduplicates data held within?

Background I have backups of a website which stores all of it's data into a single file. This file is several gigs large and I have many different backups of this file. Most of the data within is ...
9
votes
4answers
2k views

Is there a compression or archiver program for Windows that also does deduplication?

I'm looking for an archiver program that can perform deduplication (dedupe) on the files being archived. Upon unpacking the archive, the software would put back any files it removed during the ...
1
vote
2answers
283 views

Find unique files across 2 drives

I'm trying to compare 2 volumes, which mainly have the same content, but in completely different folder structures. (Around 2 million files.) I want to see which files are ONLY in 1 folder, not the ...
0
votes
1answer
108 views

Deduplicating Multiple File Types Across Multiple Sources

We are faced with a situation where data has been backed-up to several external mediums and we are undergoing an exercise to consolidate the data. The data is comprised of binary files, audio, video, ...
0
votes
1answer
741 views

avoid duplicate files during backup

I'm backing up 2 drives to an external backup drive. The 2 source drives have a lot of duplicate files (name/size) between them (for work). One of the source drives is already backed up to the ...
8
votes
6answers
3k views

Ways to deduplicate files

I want to simply backup and archive the files on several machines. Unfortunately, the files have some large files that are the same file but stored differently on different machines. For instance, ...
2
votes
3answers
445 views

Backup of 5TB server

I'm looking for a software that will help me backup a server with 5TB of data. My wishlist: Backup on 1-2TB SATA disks Should need little to no configuration (just the directories to backup and the ...
4
votes
4answers
4k views

Is there a diff utility that allows you to exclude columns?

For example, I have a text file, each line is a long string. I want to exclude 2 "segments" of this string, say columns 1-7 and 20-22. So the bottom 2 lines below would be a match: ...
1
vote
2answers
649 views

Windows 7 file-based backup service [closed]

I'm looking for a good replacement for Lazy Mirror, since it doesn't support Windows 7 well. Pros: One of the things I really loved about Lazy Mirror is that it always maintains a "full" backup, but ...
1
vote
1answer
270 views

Safety of redirecting System folders (e.g. : program files ,users,… ) using symbolic links?

Can redirecting system folders as mentioned above using symbolic links cause problems to the system or some applications? And is there any problem if I redirect the folders to another partition on ...