Deduplication is the finding and removal of duplicate files.

learn more… | top users | synonyms (1)

3
votes
1answer
34 views

How to replace all duplicate files with hard links?

I have two folders containing various files. Some of the files from the first folder have an exact copy in the second folder. I would like to replace those with a hard link. How can I do that?
0
votes
0answers
11 views

Receive only 1 message when I'm listed as the recipient in 2 ways (Outlook 2013)

As a result of the following scenario, two identical emails appear in my Outlook 2013 inbox: to: my_email@my_work_domain.com cc: email_distribution_list@my_work_domain.com (which my email is ...
0
votes
0answers
27 views

Is in-line data de-duplication worthwhile when backing up for a photographer?

I create RAW image files plus a small selection of JPEG files derived from the RAW masters. The JPEGs once created are very, very rarely edited again but when they are because they are recompressed, ...
2
votes
1answer
18 views

How do I use an image multiple times in a OneNote document while storing the image only once?

I'm using OneNote 2013 for family history. I have created a page for each person with photos of and documents about that person. In several cases I have group photos with many family members in ...
1
vote
1answer
844 views

Efficient removal of duplicate records across multiple Excel sheets

I have a workbook with 2 sheets. Sheet 1 has 3 columns (1A, 1B and 1C) and sheet 2 has 2 columns (2A and 2B). Now I need to delete rows from sheet 1 if 1A and 1B match any row in sheet 2 (i.e. 1A=2A ...
0
votes
1answer
33 views

Is it possible to perform Deduplication recursively?

If anybody is familiar with the concept of Deduplication (If not,read link on wikipedia) http://en.wikipedia.org/wiki/Data_deduplication Is it possible to perform Deduplication recursively i.e. ...
1
vote
1answer
42 views

In Windows, is there any software to prevent duplication of files in a folder?

Let's say I have two files c:\f1\data.txt and c:\f2\data.txt. If I give the command copy c:\f2\data.txt c:\f1\data\ I will receive a warning about the potential owerwriting. I will receive the ...
0
votes
2answers
38 views

Virtual folder combining a read-only and read-write directory (file deduplication)

Is it possible in linux to have a directory that contains all the contents of another directory (in a read-only fashion) but you are able to write to the directory without it affecting the 'parent' ...
0
votes
0answers
32 views

Is it possible to deduplicate a mounted truecrypt volume using Windows 2012?

An external volume has been encrypted using Truecrypt. Is it possible to run deduplication using Windows 2012 once the volume has been mounted?
17
votes
3answers
476 views

How to deduplicate 40TB of data?

I've inherited a research cluster with ~40TB of data across three filesystems. The data stretches back almost 15 years, and there are most likely a good amount of duplicates as researchers copy each ...
1
vote
1answer
101 views

Deleting all but latest revision of a file

So I have these massive lists of drawings that I've done for work and I'd like to be able to dump them all into one folder and run a batch that would delete all of the older revs and leave the highest ...
0
votes
1answer
227 views

ZFS + dedup: copy many small files fast

I'm running a ZFS backup server (with deduplication) to backup websites and store them with retention. To do this, I transfer all files to the backup server during the first backup, and make a copy of ...
0
votes
3answers
10k views

How to remove duplicate row based on some columns

I have a excel sheet which contains duplicate I want to remove a row if its A C D E F columns are same (Ignore B while calculating duplicates but remove it while removing a row) At the moment it ...
0
votes
1answer
83 views

Deduplicating Multiple File Types Across Multiple Sources

We are faced with a situation where data has been backed-up to several external mediums and we are undergoing an exercise to consolidate the data. The data is comprised of binary files, audio, video, ...
0
votes
2answers
77 views

Duplicate Deletion Script does nothing

An associate of mine wrote a script for me that should theoretically delete files that have the same name, regardless of their file type. Specifically, it checks mp3, m4a, wma, wav files. However, ...
1
vote
2answers
194 views

Find unique files across 2 drives

I'm trying to compare 2 volumes, which mainly have the same content, but in completely different folder structures. (Around 2 million files.) I want to see which files are ONLY in 1 folder, not the ...
2
votes
2answers
856 views

Remove duplicate lines from a very large text-file [duplicate]

I have a very large text-file (> 50 GB), but most lines are duplicate, so I want to remove them. Is there any way to remove duplicates lines from a file, and handle files > 2GB? Because every method I ...
1
vote
1answer
211 views

Safety of redirecting System folders (e.g. : program files ,users,… ) using symbolic links?

Can redirecting system folders as mentioned above using symbolic links cause problems to the system or some applications? And is there any problem if I redirect the folders to another partition on ...
3
votes
3answers
73 views

Is there such a thing as a file hosted container which deduplicates data held within?

Background I have backups of a website which stores all of it's data into a single file. This file is several gigs large and I have many different backups of this file. Most of the data within is ...
0
votes
2answers
4k views

How to Enable Data Deduplication in Windows 8.1 client

How can i enable Data Deduplication in Windows 8.1 x64 client. http://weikingteh.wordpress.com/2013/01/15/how-to-enable-data-deduplication-in-windows-8/ The above link discusses the same for a ...
-1
votes
1answer
657 views

Duplicate video finder? [duplicate]

Is there such a thing as a duplicate video finder that will look for duplicates based on video/audio comparisons of files rather than just the file size?
8
votes
4answers
2k views

How can I have two files with the same name in a directory when mounted with NFS?

I have a C++ application test that creates 10,000 files in an NFS mounted directory, but my test recently failed once due to one file appearing twice with the same name in that directory with all the ...
2
votes
2answers
624 views

Deduplicate files using the command line in OSX Lion

I am using OSX 10.8.4 and would like to remove duplicate files in a single flat directory if an md5 hash matches. I have looked up several answers but none of them work for me, I think because of ...
0
votes
2answers
73 views

How can I locate duplicate binary files using freely available tools?

As part of an exercise to help reduce the duplication of files, management would like to get a report of all duplicate binary files including images as these seems to exceed 100,000 or more and ...
0
votes
1answer
376 views

Deduplication efficiency with TrueCrypt File container and crashplan

I have a 100GB truecrypt container file (among lots of other things) that I am backing up with crashplan onto an external HDD. I have disabled "Preserve modification timestamp of file containers" so ...
0
votes
0answers
132 views

command line extracting especified file from 7z file

I would like to extract a specific file from a 7z file, but that filename exist twice on different directories. How can I choose one of them? I tried using compressed path but I received an error ...
0
votes
0answers
17 views

Background way to find duplicate media files [duplicate]

Any know a way to semi-automatically (in background?) identify (exact) duplicate media files to simplify deduplication? Media files eat up server space yet the need ot avoid loss leads to multiple ...
1
vote
1answer
402 views

What is the best way to merge two sets of files, but for duplicates to select the older file in Windows?

I have stumbled into a dumb usage case. I want to merge two folders in a particular way (I guess these are called folders now, I still think of them as directories, but whatever). The folders have ...
2
votes
2answers
403 views

Deduplicate Files inside multiple .tar.gz archives

I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB). I would like to archive these files and deduplicate (solid compression) files ...
0
votes
0answers
99 views

Copy only files that don't exist regardless of file name?

My dropbox account apparently renames all auto uploaded images. I'm about to clear my phone and I would like to verify that all images are in the dropbox folder before deleting them. If they had their ...
0
votes
1answer
510 views

avoid duplicate files during backup

I'm backing up 2 drives to an external backup drive. The 2 source drives have a lot of duplicate files (name/size) between them (for work). One of the source drives is already backed up to the ...
0
votes
1answer
731 views

How to identify duplicate files (by name and size) and delete them? [duplicate]

Possible Duplicate: Find duplicate files on Mac OS X I am trying to grep duplicate files in this selection: find . -name '*.CR2' I want to delete duplicate CR2 on my RAW files archive. ...
0
votes
1answer
81 views

Checking files on an external drive to whole computer for duplicates

I'm looking to use an external drive for a new purpose and want to see which, if any, files on it are not already on my main pc. I've seen comparison programs that can compare folders or one drive to ...
3
votes
1answer
60 views

Ensure the download is not a dupe

I want to (recursively) download a directory of files to a location from a WebDav Server. If the file is alreay present (somewhere there), it shall not be downloaded again. However the folder ...
1
vote
1answer
4k views

View ZFS deduplication ratio on a dataset

I have a tank consisting of several datasets, only one of which is configured to use deduplication. How can I see the ratio for this dataset? I get a ratio of 1.00x for the whole pool but I imagine ...
1
vote
4answers
538 views

Duplicate name exists solution

I have about 70 pc's with exactly same hardware. I decided to automate turning on and off. I took 1 PC. Here is what I've done: Changed bios configuration so that now pc's waking when I turn on AC ...
2
votes
1answer
682 views

ZFS compressed and deduped filesystem on file

I can create squashed filesystems with mksquashfs in this way: mksquashfs dir-1 dir-2 ... dir-n filesystem-on-file.squash -comp xz -Xbcj x86 -Xdict-size 1048576 -b 1024k mksquashfs provides ...
3
votes
3answers
539 views

Fuzzy Duplicate File Finder?

Is there any GUI application (either for Windows or Linux) that implements fuzzy hashing algorithms like ssdeep or sdhash to find near duplicate files (mostly text files)?
4
votes
1answer
791 views

How to find duplicate images and compare their meta-data?

Is there an application for Windows that finds duplicate images and lets quickly preview the image and, what is most important, the meta-data of the image files. I can see there are programs for ...
6
votes
3answers
2k views

Why do I have identical files in one directory, in Windows 7?

I just rebuilt my system after a new Power supply fired my CPU, MBD, Video card & a Blu-Ray drive. During all this I had to restore Windows from a backup (sector copy). All seems well, but ...
4
votes
3answers
3k views

iTunes - One track in several albums

I'm using Windows 7 to do this. I have quite an extensive library of music and now I have quite a lot of duplicate tracks but in different albums. Obviously I don't really want duplicate files but I'd ...
3
votes
1answer
115 views

Painless way of consolidating files across multiple machines/OSes

Just bought a NAS. So I thought I'd get all our photos, media files and pdfs consolidated, de-duplicated, de-junked and virus-checked and stick them all on it. We have 3 laptops, one running Windows, ...
0
votes
1answer
1k views

WMP 12.0 adds duplicate entries for same file mapped network drive

Windows Media Player 12 is adding duplicate entries for each file on my mapped network drive. When I try and rebuild the index by going to %UserProfile%\Appdata\Local\Microsoft\Windows Media, I am ...
1
vote
0answers
427 views

How to deduplicate files when storing in the SDFS file system using OpenDedup

I followed the link, http://www.opendedup.org/quickstart for installing OpenDedup in my linux machine (Ubuntu 12.04 - 64 bit). I created a SDFS volume and mounted it. I copied two similar Ubuntu VMs, ...
1
vote
2answers
492 views

Program for remove exact duplicate files while caching search results

We need a Windows 7 program to remove/check the duplicates but our situation is somewhat different than the standard one for which there are enough programs. We have a fairly large static archive ...
2
votes
1answer
316 views

How to de-dupe identical photos that have a slightly different file size?

I imported many photos using the new "camera import" feature of Dropbox. Many of those were duplicates of photos previously imported by direct copying from the camera. Strangely, the Dropbox import ...
1
vote
1answer
228 views

Is there a way to remove duplicate emails from a remote account?

I have a user who has multiple duplicated emails across multiple folders on his IMAP account. How he managed to create them is beyond me, but mine is not to reason why, just to fix it! Can anyone ...
3
votes
0answers
266 views

Rsync, how can I delete remote duplicates?

I'm trying to delete remote duplicate files without transferring the non-existing files, and without updating the existing files. If I specify both --existing and --ignore-existing (along with -av ...
2
votes
2answers
335 views

Duplicate File Scanner

I have a 15TB storage network, and I am down to about 2.5TB now (due to a large amount of duplicates). I have tried many scanners, but I have had little success, eventually they all crash due to the ...
0
votes
1answer
222 views

Find duplicates of a particular file

I downloaded an icon library from some website and I used several of the icons in a project. Now I need to find another icon from the library, but I can not find the library on my computer. Because I ...