Deduplication is the finding and removal of duplicate files.

learn more… | top users | synonyms (1)

0
votes
2answers
3k views

How to Enable Data Deduplication in Windows 8.1 client

How can i enable Data Deduplication in Windows 8.1 x64 client. http://weikingteh.wordpress.com/2013/01/15/how-to-enable-data-deduplication-in-windows-8/ The above link discusses the same for a ...
0
votes
0answers
13 views

Is there a diff-like tool that searches the directory for existence of the file or folder?

In an attempt to de-duplicate the data on my system, I've used tools such as Gemini for finding duplicate files across directories and diff tools like Kaleidescope for understanding how the ...
2
votes
1answer
364 views

Deduplicate files using the command line in OSX Lion

I am using OSX 10.8.4 and would like to remove duplicate files in a single flat directory if an md5 hash matches. I have looked up several answers but none of them work for me, I think because of ...
0
votes
1answer
60 views

ZFS + dedup: copy many small files fast

I'm running a ZFS backup server (with deduplication) to backup websites and store them with retention. To do this, I transfer all files to the backup server during the first backup, and make a copy of ...
0
votes
0answers
48 views

Deduplication vs Incremental Backups

I need good arguments for that question(s): Why would you take a daily full backup and deduplicate the volume, instead of taking a single full backup and afterwards continuing with incremental ...
7
votes
4answers
1k views

Is there a compression or archiver program for Windows that also does deduplication?

I'm looking for an archiver program that can perform deduplication (dedupe) on the files being archived. Upon unpacking the archive, the software would put back any files it removed during the ...
0
votes
0answers
25 views

mv backup but instead of creating tilde directories, create tilde files in subdirectories?

I have 2 deep directory structures that are (probably) identical and I want to end up with a single one, but not auto-resolve existing conflicts (I'm guessing rsync would clobber something). I just ...
0
votes
3answers
101 views

How to remove duplicate row based on some columns

I have a excel sheet which contains duplicate I want to remove a row if its A C D E F columns are same (Ignore B while calculating duplicates but remove it while removing a row) At the moment it ...
1
vote
2answers
122 views

Find unique files across 2 drives

I'm trying to compare 2 volumes, which mainly have the same content, but in completely different folder structures. (Around 2 million files.) I want to see which files are ONLY in 1 folder, not the ...
0
votes
0answers
35 views

Copy files to location, skipping duplicates (in source and dest. location)

I have an external hardrive I think is running down breaking (unable to delete some files from it). I copied a bunch of files over already but ran out of space. How do I dedup while copying/How to I ...
0
votes
1answer
41 views

Deduplicating Multiple File Types Across Multiple Sources

We are faced with a situation where data has been backed-up to several external mediums and we are undergoing an exercise to consolidate the data. The data is comprised of binary files, audio, video, ...
0
votes
2answers
65 views

Duplicate Deletion Script does nothing

An associate of mine wrote a script for me that should theoretically delete files that have the same name, regardless of their file type. Specifically, it checks mp3, m4a, wma, wav files. However, ...
0
votes
1answer
251 views

avoid duplicate files during backup

I'm backing up 2 drives to an external backup drive. The 2 source drives have a lot of duplicate files (name/size) between them (for work). One of the source drives is already backed up to the ...
8
votes
6answers
3k views

Ways to deduplicate files

I want to simply backup and archive the files on several machines. Unfortunately, the files have some large files that are the same file but stored differently on different machines. For instance, ...
2
votes
3answers
395 views

Backup of 5TB server

I'm looking for a software that will help me backup a server with 5TB of data. My wishlist: Backup on 1-2TB SATA disks Should need little to no configuration (just the directories to backup and the ...
3
votes
4answers
2k views

Is there a diff utility that allows you to exclude columns?

For example, I have a text file, each line is a long string. I want to exclude 2 "segments" of this string, say columns 1-7 and 20-22. So the bottom 2 lines below would be a match: ...
0
votes
0answers
63 views

How to stop Deduplication in Windows 2012 R2

Deduplication has been enabled on my server and I would like to stop it, since I don't need it to run. I have disabled the deduplication on the drive, but the deduplication process continues. How can ...
1
vote
2answers
593 views

Windows 7 file-based backup service

I'm looking for a good replacement for Lazy Mirror, since it doesn't support Windows 7 well. Pros: One of the things I really loved about Lazy Mirror is that it always maintains a "full" backup, but ...
2
votes
2answers
498 views

Remove duplicate lines from a very large text-file [duplicate]

I have a very large text-file (> 50 GB), but most lines are duplicate, so I want to remove them. Is there any way to remove duplicates lines from a file, and handle files > 2GB? Because every method I ...
1
vote
1answer
122 views

Safety of redirecting System folders (e.g. : program files ,users,… ) using symbolic links?

Can redirecting system folders as mentioned above using symbolic links cause problems to the system or some applications? And is there any problem if I redirect the folders to another partition on ...
2
votes
3answers
52 views

Is there such a thing as a file hosted container which deduplicates data held within?

Background I have backups of a website which stores all of it's data into a single file. This file is several gigs large and I have many different backups of this file. Most of the data within is ...
4
votes
1answer
1k views

Remove duplicate emails (Outlook 2003)

Sometimes our POP server has a fit and it re-sends a bunch of e-mail messages again to a particular user and I have to clean up those duplicate messages that ended up in the local outlook store. Is ...
6
votes
3answers
2k views

Why do I have identical files in one directory, in Windows 7?

I just rebuilt my system after a new Power supply fired my CPU, MBD, Video card & a Blu-Ray drive. During all this I had to restore Windows from a backup (sector copy). All seems well, but ...
-1
votes
1answer
344 views

Duplicate video finder? [duplicate]

Is there such a thing as a duplicate video finder that will look for duplicates based on video/audio comparisons of files rather than just the file size?
8
votes
4answers
1k views

How can I have two files with the same name in a directory when mounted with NFS?

I have a C++ application test that creates 10,000 files in an NFS mounted directory, but my test recently failed once due to one file appearing twice with the same name in that directory with all the ...
0
votes
2answers
65 views

How can I locate duplicate binary files using freely available tools?

As part of an exercise to help reduce the duplication of files, management would like to get a report of all duplicate binary files including images as these seems to exceed 100,000 or more and ...
0
votes
1answer
313 views

Deduplication efficiency with TrueCrypt File container and crashplan

I have a 100GB truecrypt container file (among lots of other things) that I am backing up with crashplan onto an external HDD. I have disabled "Preserve modification timestamp of file containers" so ...
0
votes
0answers
27 views

Linux delete duplicates and specify directory under which will be perserved

Is there any de-duplication Linux software which allows specify folder under which duplicates will be found and immediately deleted (but one copy of duplicates will be preserved) and also allows ...
0
votes
0answers
94 views

command line extracting especified file from 7z file

I would like to extract a specific file from a 7z file, but that filename exist twice on different directories. How can I choose one of them? I tried using compressed path but I received an error ...
3
votes
3answers
335 views

Fuzzy Duplicate File Finder?

Is there any GUI application (either for Windows or Linux) that implements fuzzy hashing algorithms like ssdeep or sdhash to find near duplicate files (mostly text files)?
0
votes
0answers
17 views

Background way to find duplicate media files [duplicate]

Any know a way to semi-automatically (in background?) identify (exact) duplicate media files to simplify deduplication? Media files eat up server space yet the need ot avoid loss leads to multiple ...
1
vote
1answer
229 views

What is the best way to merge two sets of files, but for duplicates to select the older file in Windows?

I have stumbled into a dumb usage case. I want to merge two folders in a particular way (I guess these are called folders now, I still think of them as directories, but whatever). The folders have ...
48
votes
15answers
12k views

Which duplicate files and folders finders exist for Windows? [closed]

I need a free duplicate file finder/remover app, with ability to find duplicate files/folders by name and/or by size and to remove one of duplicates.
5
votes
4answers
237 views

Locate near-duplicate files in a folder

I have a folder full of files, with a number of duplicate files in it. Unfortunately, in a number of cases, one version is an updated version of another, so a straight byte-match doesn't locate the ...
2
votes
1answer
263 views

Deduplicate Files inside multiple .tar.gz archives

I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB). I would like to archive these files and deduplicate (solid compression) files ...
0
votes
0answers
92 views

Copy only files that don't exist regardless of file name?

My dropbox account apparently renames all auto uploaded images. I'm about to clear my phone and I would like to verify that all images are in the dropbox folder before deleting them. If they had their ...
0
votes
1answer
567 views

How to identify duplicate files (by name and size) and delete them? [duplicate]

Possible Duplicate: Find duplicate files on Mac OS X I am trying to grep duplicate files in this selection: find . -name '*.CR2' I want to delete duplicate CR2 on my RAW files archive. ...
0
votes
1answer
71 views

Checking files on an external drive to whole computer for duplicates

I'm looking to use an external drive for a new purpose and want to see which, if any, files on it are not already on my main pc. I've seen comparison programs that can compare folders or one drive to ...
3
votes
1answer
59 views

Ensure the download is not a dupe

I want to (recursively) download a directory of files to a location from a WebDav Server. If the file is alreay present (somewhere there), it shall not be downloaded again. However the folder ...
0
votes
1answer
532 views

md5deep not working with duplicated files in different directories

I want to use md5deep to check that some copied directories are copied correctly. But seems md5deep doesn't use the filenames stored in the genearted .md5-files when matching! This actually means, ...
4
votes
1answer
523 views

How to find duplicate images and compare their meta-data?

Is there an application for Windows that finds duplicate images and lets quickly preview the image and, what is most important, the meta-data of the image files. I can see there are programs for ...
1
vote
1answer
3k views

View ZFS deduplication ratio on a dataset

I have a tank consisting of several datasets, only one of which is configured to use deduplication. How can I see the ratio for this dataset? I get a ratio of 1.00x for the whole pool but I imagine ...
1
vote
4answers
340 views

Duplicate name exists solution

I have about 70 pc's with exactly same hardware. I decided to automate turning on and off. I took 1 PC. Here is what I've done: Changed bios configuration so that now pc's waking when I turn on AC ...
1
vote
1answer
185 views

Check for duplicate files in two collections, ignoring duplicates within A

I have four similar large collections of website files. They are archived copies of a similar directory tree from different times in the past. I want to merge them all to the most recent collection, ...
1
vote
3answers
402 views

Program for remove exact duplicate files while caching search results

We need a Windows 7 program to remove/check the duplicates but our situation is somewhat different than the standard one for which there are enough programs. We have a fairly large static archive ...
1
vote
1answer
214 views

Is there a way to remove duplicate emails from a remote account?

I have a user who has multiple duplicated emails across multiple folders on his IMAP account. How he managed to create them is beyond me, but mine is not to reason why, just to fix it! Can anyone ...
4
votes
2answers
706 views

Software to detect mp3 almost-duplicates?

Because of some unfortunate circumstances I noticed I irrevertibly mixed up my sorted and retagged mp3s with an old backup. That means now I have files that are basically duplicates except for the id3 ...
2
votes
1answer
289 views

How to de-dupe identical photos that have a slightly different file size?

I imported many photos using the new "camera import" feature of Dropbox. Many of those were duplicates of photos previously imported by direct copying from the camera. Strangely, the Dropbox import ...
2
votes
1answer
494 views

ZFS compressed and deduped filesystem on file

I can create squashed filesystems with mksquashfs in this way: mksquashfs dir-1 dir-2 ... dir-n filesystem-on-file.squash -comp xz -Xbcj x86 -Xdict-size 1048576 -b 1024k mksquashfs provides ...
1
vote
0answers
209 views

Duplicate Images / Tags - sorting and removing

I have folders full of camera images that I have copied and backed up between my laptop, computer and external hard drive(s). I want to organise and sort them, and remove duplicates, so I have used a ...