Questions tagged [deduplication]

Deduplication is the finding and removal of duplicate files, or to keep both files but share the same allocation unit on the storage medium

Filter by
Sorted by
Tagged with
1 vote
1 answer
5k views

How to safely consolidate duplicate folders on Google Drive and then delete the emptied ones, syncing the kept ones?

I was reinstalling my system, and I carelessly copied folders to the folder I use for Google Drive and started syncing at the same time. That resulted in my files being split between duplicate ...
37 votes
6 answers
17k views

How to replace all duplicate files with hard links?

I have two folders containing various files. Some of the files from the first folder have an exact copy in the second folder. I would like to replace those with a hard link. How can I do that?
0 votes
2 answers
169 views

Is there a way to select the first instance of a duplicated file in dupeguru?

dupeguru has helped me find duplicated files, but it only allows me to delete the second instance of duplicated files (as shown in the screenshot). I wonder if there's a way to delete the first ...
2 votes
1 answer
48 views

Can I control which file jdupes preserves when using hardlinks?

The jdupes deduplication program allows you to create hardlinks, so that the files are not deleted, but linked together so that the disk space is used only once for multiple copies. Is there a way to ...
-1 votes
5 answers
3k views

Why do duplicate files have different checksums?

I am trying to remove hundreds of duplicate .PST files from a single folder containing archived .PST files from Outlook. I've tried a couple of Duplicate Remover apps, which rely on checksums to ...
59 votes
15 answers
30k views

Which duplicate files and folders finders exist for Windows? [closed]

I need a free duplicate file finder/remover app, with ability to find duplicate files/folders by name and/or by size and to remove one of duplicates.
0 votes
0 answers
117 views

Remove but 'remember' (to later restore) duplicate frames from video/ image sequence?

I'm upscaling animated content (via ChaiNNer) in which a number of frames (often 25-50%) are duplicates, wasting significant gpu time; I've been attempting to remove duplicates pre upscale but have ...
1 vote
2 answers
763 views

Can I find duplicate files with the same path in different locations?

I have two folders/directories: C:\MyData and C:\MyDataBackup and the person that owns those two folders/directories and does not remember if they have edited the files in the original or in the ...
1 vote
1 answer
127 views

How to remove duplicated text on every line using Notepad++?

I want to remove duplicated words in each line using Notepad++. Example: Flooring Services, Carpet, Flooring Services, Tile, Flooring Services In the above, Flooring Services is repeated 3x. I only ...
0 votes
1 answer
535 views

Removing duplicate rows in a CSV file as compared to another CSV file

I have a CSV file named CSV1 with over 5000 rows and 3 columns - last name, first name and email address. I have another file named CSV2 with about 2700 rows with the same 3 columns. I have to remove ...
5 votes
3 answers
7k views

Digikam (KDE) - How to remove all duplicate photos, but keep the reference images?

Though Digikam provides a nice feature of finding duplicate photos, it does not, for what I can see, make it easy to delete duplicates and keep original photos. I hope that the function is somewhere ...
3 votes
4 answers
655 views

Backup of 5TB server

I'm looking for a software that will help me backup a server with 5TB of data. My wishlist: Backup on 1-2TB SATA disks Should need little to no configuration (just the directories to backup and the ...
1 vote
1 answer
515 views

Remove email (.eml) duplicates

I have a folder with around 50.000 emails in .eml format. There are many duplicates, even triplets or quadriplets of them, i suppose around 30.000 in total. I have tried to remove duplicates using ...
12 votes
4 answers
106k views

How to remove duplicate rows based on some columns

I have an Excel sheet which contains duplicate rows. I want to remove a row if its values in columns A C D E F are same as another row's values in those columns (ignore column B while calculating ...
0 votes
0 answers
93 views

ZFS deduplication table not paged out to RAM

I have a pool on which there is 3.41 TB of data, deduplication is enabled on the volumes, judging by the output of the zpool status -D <pool_name> ... dedup: DDT entries 73665285, size 696B on ...
2 votes
1 answer
5k views

How to find duplicate id entries in a xml file

How can I quickly narrow down duplicate id entries in a xml file, example: <entry id="A">... <entry id="B">... <entry id="A">... and output them id="A" dup 2 times Just to let ...
0 votes
1 answer
300 views

btrfs fi du: several sets of shared extents

I have a lot of files in various directories on a btrfs filesystem. They are all identical, and some may have been created using a "old-style" full copy, some are clones (cp --reflink). I ...
1 vote
0 answers
367 views

How to overwrite all files only if SAME size and exact duplicate while renaming if different size?

I'm using Teracopy, but Windows also does not have this function. Instead it wants me to replace the file only if DIFFERENT, or skip it if the name is the same. I want to do the opposite. I need to ...
11 votes
1 answer
628 views

In bash, how to find all copies of a given file in particular directories?

Let's say we have a file /a_long_path_1/foo.doc of size, say, 12345 bytes, and we would like to find all copies of this file in directories /a_long_path_2 and /a_long_path_3 including all their ...
0 votes
0 answers
2k views

Cannot install Deduplication role on Windows Server 2019 0x80073701

I want to install the Deduplication role on Windows Server 2019 but I always get an 0x80073701 Error. I tried all sfc and Dism, updates but there where no problems, I also reinstalled update ...
0 votes
0 answers
183 views

How can I look for duplicate files from different base folders in bash?

I know fdupes, which can find duplicate files from various directories I provide it: fdupes -r <directoryA> <directoryB> <...> But this seems to “throw all files into the same ...
1 vote
1 answer
1k views

Remove duplicate files within specific directories, keeping them in another with Bash

I've seen a few questions on here regarding removing duplicate files in various circumstances but I can't find anything along the lines of what I'm trying to accomplish. The closest I came was seeing ...
0 votes
2 answers
446 views

File Deduplication and Filtering

I had to run a TestDisk file recovery software on one of my drives. The result was a little over 16,000 files who's names are basically an alpha numeric string. Many of these files contain duplicate ...
3 votes
4 answers
4k views

Deduplicate Files inside multiple .tar.gz archives

I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB). I would like to archive these files and deduplicate (solid compression) files ...
11 votes
7 answers
6k views

Ways to deduplicate files

I want to simply backup and archive the files on several machines. Unfortunately, the files have some large files that are the same file but stored differently on different machines. For instance, ...
0 votes
1 answer
202 views

Detect files with multiple extensions

I've a synology running a kind of linux OS. I've transfered there a lot of photos. Unfortunately, some are with .jpg and some with .JPG depending of the tool I used to import. These are considered as ...
0 votes
0 answers
51 views

delete files from folder and subfolders named filename - copy

I have had an issue with Microsoft OneDrive and now I have many files named filename - copy. I found This Post which says to use del "* - Copy.*" which works, within a given directory, but ...
0 votes
1 answer
901 views

Cleanup duplicate mails after fetchmail messed up the fetchids list

I use fetchmail since years, to get my mails from a free private email provider, always with the "keep" option. Now, due to an update in mailu, the fetchids file got messed up and I have ...
2 votes
2 answers
120 views

Find and delete near duplicate lines with certain punctuation

I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
4 votes
4 answers
5k views

iTunes - One track in several albums

I'm using Windows 7 to do this. I have quite an extensive library of music and now I have quite a lot of duplicate tracks but in different albums. Obviously I don't really want duplicate files but I'd ...
3 votes
3 answers
3k views

ZFS dedup performance optimization

I'm running a VDI server. Many of the VDI users are installing the same apps on their desktops, so zfs set dedup=on vdipool/myDesktop comes in handy. How can I optimize the performance impact of zfs ...
3 votes
1 answer
220 views

Which duplicate should I remove?

A while ago I somehow managed to duplicate a lot of photos but didn't realise till many months later and now don't know which one is the duplicate. For the duplicates, there is a common pattern ...
1 vote
1 answer
1k views

Removing Duplicate Files on MacOS Terminal Using MD5

I found the following command that makes use of md5 file hashes to find duplicate files and list them in a text file. find ./ -type f -exec md5 {} \; | awk -F '=' '{print $2 "\t" $1}' | sort ...
0 votes
1 answer
2k views

The Remove Duplicate Messages add on to Mozilla Thunderbird does not find all duplicates

I have upgraded my email hosting and copied bulk messages from folders in the old server to the same folder in the new server. However in some cases I have copied many messages multiple times so have ...
1 vote
1 answer
547 views

How to delete all duplicates of a file linked by CoW on BTRFS

The glory of Copy-on-Write (CoW) allows us to copy a file as many times as we like, without duplicating the used space. I have used this functionality extensively (as well as manual deduplication of ...
1 vote
0 answers
132 views

Music managing software delete lower quality duplicates [closed]

Until the end of the last year I was one of the few people who used Google Play Music, and as it was discontinued, I downloaded all my music using Google Takeout. Now I have almost all of my songs in ...
1 vote
1 answer
69 views

Compare only files within the same source folder vs. Compare only files within the same source and sub folder

Isn't 2 and 4 identical? Can you please distinguish them? Subfolders must be within the source folder. Then "within the same source folder" in 2 must include all subfolders. Compare files ...
0 votes
1 answer
1k views

Explain like I'm 5 these 9 methods to compare similar pictures – Hash, MD5, SHA?

Please compare and contrast – like I'm 5, in SIMPLE English – all 9 methods below? English isn't my first language.
0 votes
0 answers
209 views

Anti-Twin's Byte by Byte" vs. "Compare images (pixels)"

Kindly see the red rectangle below. How does "Byte by Byte" differ from "Compare images (pixels)"? Which is more accurate? What are the pros and cons of each? Advise me if I got ...
41 votes
5 answers
40k views

Is there a way to extract duplicate lines in Sublime Text?

I need to perform 2 operations in Sublime Text: extract unique lines and extract duplicate lines. For example for input a b a Extract duplicates should result in: a and Extract unique should ...
0 votes
1 answer
514 views

Best way to find duplicate data files between two Linux machines?

I have two Debian boxes at home that have various photos, audio, video and other data files on them. They are both disorganised, I'd like to organise them and remove duplicate files between them. ...
14 votes
7 answers
6k views

What is the best method to remove duplicate image files from your computer?

I have a lot of duplicate image files on my Windows computer, in different subfolders and with different file names. What Python script or freeware program would you recommend for removing the ...
2 votes
1 answer
90 views

Remove files in Directory B that also exist in Directory A

This page has tells how to remove files in Directory B that also exist in Directory A. I want to use the windows solution but without the copying. Can someone tell me how to edit the FOR line to ...
1 vote
2 answers
2k views

Find duplicate hashes

I have a bunch of files (4800 files), that I believe contain many duplicates and triples. Problem is that they don't also have the same names. I already weeded out the easy ones with same names. The ...
0 votes
0 answers
22 views

Why are the contents of my Downloads folder duplicating to my account folder?

I just went to my C:/Users/[username] folder and noticed that every file that is in my downloads folder is also in my [username] folder. Why is that happening and how can I get this to stop?
0 votes
1 answer
93 views

Looking for a way to remove duplicate entries with variable information

I am trying to remove duplicates having variable information. Column F = delete time Column Q = recipient code Column R = recipient address I need to remove duplicates that match on row ...
0 votes
0 answers
171 views

How to deduplicate Virtualbox VMs

I'm generally creating a new clone of my existing VM with "linked clone" option to save space. However, for some reasons (backup/restore etc.), I end up with fully cloned machines thus using huge ...
0 votes
1 answer
90 views

Removing duplicates

i've got plenty of those duplicates: 2017-11-22_3083_smokey_brights_-_come_to_terms.mp3 2017-12-01-Smokey_Brights_-_Come_To_Terms.mp3 How can I keep only one of each ? Unfortunately, their md5sum ...
2 votes
2 answers
701 views

duplicate file/folder finder application for windows 7

I got old archives of my old computers, Now I decided to merge all these folders and organize them, I got many backups so I need app that will find duplicate files and delete / move them, any ideas?
0 votes
1 answer
2k views

WMP 12.0 adds duplicate entries for same file mapped network drive

Windows Media Player 12 is adding duplicate entries for each file on my mapped network drive. When I try and rebuild the index by going to %UserProfile%\Appdata\Local\Microsoft\Windows Media, I am ...