Deduplication is the finding and removal of duplicate files.

learn more… | top users | synonyms (1)

0
votes
0answers
10 views

How to store effectively pictures with minimal differences [on hold]

I'd like to store effectively list of images that have minimal differences (e.g. photo with small marks here and there) and I'd like to store them effectively on the disk (minimum disk space occupied) ...
0
votes
1answer
18 views

How to find extracted Zip files and their folder in mac

How can i list all zip files and their extracted folders in order to remove one of them, i downloaded lots of zip files in years(like everybody) and extracted them so now they are taking twice as ...
-1
votes
1answer
47 views

Removing duplicates from txt file

I have a txt file that can contain up to 13.000.000 randomly generated codes seperated with "\n" by the fputcsv function in php. What is the most effective way of removing duplicates from this file? ...
0
votes
1answer
31 views

What are the odds file size remains the same if you photoshop an image?

My uncle is a photographer with lots and lots of photoshoots. He stored (duplicates of) backups across a variety of external hard drives. 90% of all backups are the same, but some have had some edits, ...
0
votes
1answer
124 views

Excel: Copy all cell values from an entire sheet to a new sheet, sort and deduplicate into a single list of unique values

The data (all text values) I have is scattered across the sheet (multiple rows and columns with gaps inbetween) and it is all calculated using formulas now I would like to make sure that I know what ...
0
votes
1answer
109 views

Find duplicate media files and auto organize, on Linux

Is there an application similar to MediaMoney (on Windows), but on Linux (Debian or Ubuntu) that given when a disk-drive or a root directory, can search across it thoroughly, identifying duplicate ...
0
votes
1answer
49 views

Remove Duplicates inconsistently choosing which records to delete

So generally when using the remove duplicates function in Excel the oldest record will be kept - and that's exactly what I want to happen! But recently when running Remove Duplicates, it is ...
0
votes
1answer
39 views

How to delete one of each set of duplicate values in Excel?

I have a spreadsheet that contains many duplicates. Instead of removing all of the duplicates, I would like to remove one of each duplicate. I want to remove one record from each set of duplicates. ...
1
vote
0answers
45 views

Using btrfs cow functionallity to rapidly and efficiently subset large files

[My aim is to use btrfs cow features to subset 30GB to 300GB files into several 10GB files without producing substantial i/o nor using additional disk space. i.e.: 300GB dna_original.fastq 10GB ...
1
vote
0answers
105 views

using btrfs features to rapidly and efficiently combine large files

My aim is to use btrfs features to combine several 3GB files into a single 9GB file without producing substantial i/o nor using additional disk space. If this works for this set of data, I plan to ...
1
vote
2answers
566 views

Removing files with same name and different extension

I have a large music collection. It is organized in the following structure: artist / album / trackTitle. Over the course of time, in some directories, I have duplicate-ish files; most of my files ...
0
votes
2answers
182 views

Undo Creating Symbolic Links by copying back the original, rename the file and deleting the link

I used a windows 7 utility to delete exact byte per byte duplicate files and replace them with a symbolic link before I knew that the symbolic link type is not allowed in our corporate domain. I need ...
4
votes
3answers
1k views

How to replace all duplicate files with hard links?

I have two folders containing various files. Some of the files from the first folder have an exact copy in the second folder. I would like to replace those with a hard link. How can I do that?
0
votes
0answers
66 views

Is in-line data de-duplication worthwhile when backing up for a photographer?

I create RAW image files plus a small selection of JPEG files derived from the RAW masters. The JPEGs once created are very, very rarely edited again but when they are because they are recompressed, ...
2
votes
1answer
30 views

How do I use an image multiple times in a OneNote document while storing the image only once?

I'm using OneNote 2013 for family history. I have created a page for each person with photos of and documents about that person. In several cases I have group photos with many family members in them....
0
votes
1answer
58 views

Is it possible to perform Deduplication recursively?

If anybody is familiar with the concept of Deduplication (If not,read link on wikipedia) http://en.wikipedia.org/wiki/Data_deduplication Is it possible to perform Deduplication recursively i.e. ...
0
votes
2answers
68 views

In Windows, is there any software to prevent duplication of files in a folder? [closed]

Let's say I have two files c:\f1\data.txt and c:\f2\data.txt. If I issue the command copy c:\f2\data.txt c:\f1\data\ I will receive a warning about the potential overwriting. I will receive the ...
0
votes
2answers
83 views

Virtual folder combining a read-only and read-write directory (file deduplication)

Is it possible in linux to have a directory that contains all the contents of another directory (in a read-only fashion) but you are able to write to the directory without it affecting the 'parent' ...
17
votes
3answers
816 views

How to deduplicate 40TB of data?

I've inherited a research cluster with ~40TB of data across three filesystems. The data stretches back almost 15 years, and there are most likely a good amount of duplicates as researchers copy each ...
1
vote
1answer
364 views

Deleting all but latest revision of a file

So I have these massive lists of drawings that I've done for work and I'd like to be able to dump them all into one folder and run a batch that would delete all of the older revs and leave the highest ...
0
votes
1answer
373 views

ZFS + dedup: copy many small files fast

I'm running a ZFS backup server (with deduplication) to backup websites and store them with retention. To do this, I transfer all files to the backup server during the first backup, and make a copy of ...
5
votes
4answers
47k views

How to remove duplicate row based on some columns

I have a excel sheet which contains duplicate I want to remove a row if its A C D E F columns are same (Ignore B while calculating duplicates but remove it while removing a row) At the moment it ...
0
votes
1answer
115 views

Deduplicating Multiple File Types Across Multiple Sources

We are faced with a situation where data has been backed-up to several external mediums and we are undergoing an exercise to consolidate the data. The data is comprised of binary files, audio, video, ...
0
votes
2answers
83 views

Duplicate Deletion Script does nothing

An associate of mine wrote a script for me that should theoretically delete files that have the same name, regardless of their file type. Specifically, it checks mp3, m4a, wma, wav files. However, I'...
1
vote
2answers
318 views

Find unique files across 2 drives

I'm trying to compare 2 volumes, which mainly have the same content, but in completely different folder structures. (Around 2 million files.) I want to see which files are ONLY in 1 folder, not the ...
2
votes
2answers
1k views

Remove duplicate lines from a very large text-file [duplicate]

I have a very large text-file (> 50 GB), but most lines are duplicate, so I want to remove them. Is there any way to remove duplicates lines from a file, and handle files > 2GB? Because every method I ...
1
vote
1answer
293 views

Safety of redirecting System folders (e.g. : program files ,users,… ) using symbolic links?

Can redirecting system folders as mentioned above using symbolic links cause problems to the system or some applications? And is there any problem if I redirect the folders to another partition on ...
3
votes
3answers
91 views

Is there such a thing as a file hosted container which deduplicates data held within?

Background I have backups of a website which stores all of it's data into a single file. This file is several gigs large and I have many different backups of this file. Most of the data within is ...
0
votes
2answers
4k views

How to Enable Data Deduplication in Windows 8.1 client

How can i enable Data Deduplication in Windows 8.1 x64 client. http://weikingteh.wordpress.com/2013/01/15/how-to-enable-data-deduplication-in-windows-8/ The above link discusses the same for a ...
-1
votes
1answer
1k views

Duplicate video finder? [duplicate]

Is there such a thing as a duplicate video finder that will look for duplicates based on video/audio comparisons of files rather than just the file size?
8
votes
4answers
4k views

How can I have two files with the same name in a directory when mounted with NFS?

I have a C++ application test that creates 10,000 files in an NFS mounted directory, but my test recently failed once due to one file appearing twice with the same name in that directory with all the ...
2
votes
2answers
1k views

Deduplicate files using the command line in OSX Lion

I am using OSX 10.8.4 and would like to remove duplicate files in a single flat directory if an md5 hash matches. I have looked up several answers but none of them work for me, I think because of ...
0
votes
2answers
87 views

How can I locate duplicate binary files using freely available tools?

As part of an exercise to help reduce the duplication of files, management would like to get a report of all duplicate binary files including images as these seems to exceed 100,000 or more and ...
0
votes
1answer
445 views

Deduplication efficiency with TrueCrypt File container and crashplan

I have a 100GB truecrypt container file (among lots of other things) that I am backing up with crashplan onto an external HDD. I have disabled "Preserve modification timestamp of file containers" so ...
0
votes
0answers
184 views

Command line extracting specified file from 7-Zip file

I would like to extract a specific file from a 7-Zip (.7z) file, but that filename exist twice on different directories. How can I choose one of them? I tried using a compressed path, but I received ...
0
votes
0answers
17 views

Background way to find duplicate media files [duplicate]

Any know a way to semi-automatically (in background?) identify (exact) duplicate media files to simplify deduplication? Media files eat up server space yet the need ot avoid loss leads to multiple ...
1
vote
1answer
621 views

What is the best way to merge two sets of files, but for duplicates to select the older file in Windows?

I have stumbled into a dumb usage case. I want to merge two folders in a particular way (I guess these are called folders now, I still think of them as directories, but whatever). The folders have ...
2
votes
3answers
606 views

Deduplicate Files inside multiple .tar.gz archives

I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB). I would like to archive these files and deduplicate (solid compression) files ...
0
votes
0answers
104 views

Copy only files that don't exist regardless of file name?

My dropbox account apparently renames all auto uploaded images. I'm about to clear my phone and I would like to verify that all images are in the dropbox folder before deleting them. If they had their ...
0
votes
1answer
855 views

avoid duplicate files during backup

I'm backing up 2 drives to an external backup drive. The 2 source drives have a lot of duplicate files (name/size) between them (for work). One of the source drives is already backed up to the ...
0
votes
1answer
869 views

How to identify duplicate files (by name and size) and delete them? [duplicate]

Possible Duplicate: Find duplicate files on Mac OS X I am trying to grep duplicate files in this selection: find . -name '*.CR2' I want to delete duplicate CR2 on my RAW files archive. How ...
0
votes
1answer
90 views

Checking files on an external drive to whole computer for duplicates

I'm looking to use an external drive for a new purpose and want to see which, if any, files on it are not already on my main pc. I've seen comparison programs that can compare folders or one drive to ...
3
votes
1answer
65 views

Ensure the download is not a dupe

I want to (recursively) download a directory of files to a location from a WebDav Server. If the file is alreay present (somewhere there), it shall not be downloaded again. However the folder ...
2
votes
1answer
6k views

View ZFS deduplication ratio on a dataset

I have a tank consisting of several datasets, only one of which is configured to use deduplication. How can I see the ratio for this dataset? I get a ratio of 1.00x for the whole pool but I imagine ...
1
vote
4answers
794 views

Duplicate name exists solution

I have about 70 pc's with exactly same hardware. I decided to automate turning on and off. I took 1 PC. Here is what I've done: Changed bios configuration so that now pc's waking when I turn on AC ...
2
votes
1answer
871 views

ZFS compressed and deduped filesystem on file

I can create squashed filesystems with mksquashfs in this way: mksquashfs dir-1 dir-2 ... dir-n filesystem-on-file.squash -comp xz -Xbcj x86 -Xdict-size 1048576 -b 1024k mksquashfs provides ...
3
votes
3answers
765 views

Fuzzy Duplicate File Finder?

Is there any GUI application (either for Windows or Linux) that implements fuzzy hashing algorithms like ssdeep or sdhash to find near duplicate files (mostly text files)?
4
votes
1answer
1k views

How to find duplicate images and compare their meta-data?

Is there an application for Windows that finds duplicate images and lets quickly preview the image and, what is most important, the meta-data of the image files. I can see there are programs for ...
7
votes
3answers
3k views

Why do I have identical files in one directory, in Windows 7?

I just rebuilt my system after a new Power supply fired my CPU, MBD, Video card & a Blu-Ray drive. During all this I had to restore Windows from a backup (sector copy). All seems well, but ...
4
votes
3answers
3k views

iTunes - One track in several albums

I'm using Windows 7 to do this. I have quite an extensive library of music and now I have quite a lot of duplicate tracks but in different albums. Obviously I don't really want duplicate files but I'd ...