Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [deduplication]

Deduplication is the finding and removal of duplicate files, or to keep both files but share the same allocation unit on the storage medium

Filter by
Sorted by
Tagged with
0 votes
0 answers
30 views

Use fdupes to compare two directories

Is it possible to use fdupes to compare the contents of two directories (dir1 and dir2), identify files in dir2 that are present in dir1, and delete those duplicates files from dir2? Alternatively, ...
Josh's user avatar
  • 143
0 votes
1 answer
56 views

In Linux with limited features (NAS box), how can I find duplicate files, then remove duplicates and substitute hardlinks to one file?

I have a NAS box running some version of Linux that I use for backing up anything and everything. It is essentially an absolute certainty that some of the files are identical duplicates. That being ...
Jim JR Harris's user avatar
0 votes
2 answers
276 views

Is there a way to select the first instance of a duplicated file in dupeguru?

dupeguru has helped me find duplicated files, but it only allows me to delete the second instance of duplicated files (as shown in the screenshot). I wonder if there's a way to delete the first ...
jeffrey's user avatar
  • 105
2 votes
1 answer
89 views

Can I control which file jdupes preserves when using hardlinks?

The jdupes deduplication program allows you to create hardlinks, so that the files are not deleted, but linked together so that the disk space is used only once for multiple copies. Is there a way to ...
allo's user avatar
  • 1,128
1 vote
1 answer
178 views

How to remove duplicated text on every line using Notepad++?

I want to remove duplicated words in each line using Notepad++. Example: Flooring Services, Carpet, Flooring Services, Tile, Flooring Services In the above, Flooring Services is repeated 3x. I only ...
Matt Lance's user avatar
0 votes
1 answer
643 views

Removing duplicate rows in a CSV file as compared to another CSV file

I have a CSV file named CSV1 with over 5000 rows and 3 columns - last name, first name and email address. I have another file named CSV2 with about 2700 rows with the same 3 columns. I have to remove ...
user1842329's user avatar
0 votes
0 answers
104 views

ZFS deduplication table not paged out to RAM

I have a pool on which there is 3.41 TB of data, deduplication is enabled on the volumes, judging by the output of the zpool status -D <pool_name> ... dedup: DDT entries 73665285, size 696B on ...
lignumq's user avatar
0 votes
0 answers
128 views

Remove but 'remember' (to later restore) duplicate frames from video/ image sequence?

I'm upscaling animated content (via ChaiNNer) in which a number of frames (often 25-50%) are duplicates, wasting significant gpu time; I've been attempting to remove duplicates pre upscale but have ...
helplessnoob's user avatar
0 votes
1 answer
361 views

btrfs fi du: several sets of shared extents

I have a lot of files in various directories on a btrfs filesystem. They are all identical, and some may have been created using a "old-style" full copy, some are clones (cp --reflink). I ...
Marian's user avatar
  • 1,118
1 vote
0 answers
407 views

How to overwrite all files only if SAME size and exact duplicate while renaming if different size?

I'm using Teracopy, but Windows also does not have this function. Instead it wants me to replace the file only if DIFFERENT, or skip it if the name is the same. I want to do the opposite. I need to ...
superuser's user avatar
  • 3,504
1 vote
2 answers
862 views

Can I find duplicate files with the same path in different locations?

I have two folders/directories: C:\MyData and C:\MyDataBackup and the person that owns those two folders/directories and does not remember if they have edited the files in the original or in the ...
Kurtibert's user avatar
  • 167
11 votes
1 answer
637 views

In bash, how to find all copies of a given file in particular directories?

Let's say we have a file /a_long_path_1/foo.doc of size, say, 12345 bytes, and we would like to find all copies of this file in directories /a_long_path_2 and /a_long_path_3 including all their ...
user avatar
0 votes
0 answers
232 views

How can I look for duplicate files from different base folders in bash?

I know fdupes, which can find duplicate files from various directories I provide it: fdupes -r <directoryA> <directoryB> <...> But this seems to “throw all files into the same ...
Bowi's user avatar
  • 1,487
1 vote
1 answer
1k views

Remove duplicate files within specific directories, keeping them in another with Bash

I've seen a few questions on here regarding removing duplicate files in various circumstances but I can't find anything along the lines of what I'm trying to accomplish. The closest I came was seeing ...
n21brown's user avatar
0 votes
2 answers
510 views

File Deduplication and Filtering

I had to run a TestDisk file recovery software on one of my drives. The result was a little over 16,000 files who's names are basically an alpha numeric string. Many of these files contain duplicate ...
Reahreic's user avatar
  • 103
0 votes
1 answer
218 views

Detect files with multiple extensions

I've a synology running a kind of linux OS. I've transfered there a lot of photos. Unfortunately, some are with .jpg and some with .JPG depending of the tool I used to import. These are considered as ...
Skippy's user avatar
  • 13
0 votes
0 answers
53 views

delete files from folder and subfolders named filename - copy

I have had an issue with Microsoft OneDrive and now I have many files named filename - copy. I found This Post which says to use del "* - Copy.*" which works, within a given directory, but ...
auxao's user avatar
  • 11
0 votes
1 answer
961 views

Cleanup duplicate mails after fetchmail messed up the fetchids list

I use fetchmail since years, to get my mails from a free private email provider, always with the "keep" option. Now, due to an update in mailu, the fetchids file got messed up and I have ...
Daniel Alder's user avatar
0 votes
0 answers
2k views

Cannot install Deduplication role on Windows Server 2019 0x80073701

I want to install the Deduplication role on Windows Server 2019 but I always get an 0x80073701 Error. I tried all sfc and Dism, updates but there where no problems, I also reinstalled update ...
Seha's user avatar
  • 1
2 votes
2 answers
125 views

Find and delete near duplicate lines with certain punctuation

I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
Priora's user avatar
  • 35
-1 votes
5 answers
4k views

Why do duplicate files have different checksums?

I am trying to remove hundreds of duplicate .PST files from a single folder containing archived .PST files from Outlook. I've tried a couple of Duplicate Remover apps, which rely on checksums to ...
RocketSurgeon's user avatar
3 votes
1 answer
239 views

Which duplicate should I remove?

A while ago I somehow managed to duplicate a lot of photos but didn't realise till many months later and now don't know which one is the duplicate. For the duplicates, there is a common pattern ...
Harry Spratt's user avatar
1 vote
1 answer
1k views

Removing Duplicate Files on MacOS Terminal Using MD5

I found the following command that makes use of md5 file hashes to find duplicate files and list them in a text file. find ./ -type f -exec md5 {} \; | awk -F '=' '{print $2 "\t" $1}' | sort ...
Train's user avatar
  • 13
0 votes
1 answer
2k views

The Remove Duplicate Messages add on to Mozilla Thunderbird does not find all duplicates

I have upgraded my email hosting and copied bulk messages from folders in the old server to the same folder in the new server. However in some cases I have copied many messages multiple times so have ...
J Collins's user avatar
  • 678
1 vote
1 answer
597 views

How to delete all duplicates of a file linked by CoW on BTRFS

The glory of Copy-on-Write (CoW) allows us to copy a file as many times as we like, without duplicating the used space. I have used this functionality extensively (as well as manual deduplication of ...
chronos00's user avatar
  • 111
1 vote
0 answers
132 views

Music managing software delete lower quality duplicates [closed]

Until the end of the last year I was one of the few people who used Google Play Music, and as it was discontinued, I downloaded all my music using Google Takeout. Now I have almost all of my songs in ...
DarkDiamond's user avatar
  • 1,879
0 votes
1 answer
1k views

Explain like I'm 5 these 9 methods to compare similar pictures – Hash, MD5, SHA?

Please compare and contrast – like I'm 5, in SIMPLE English – all 9 methods below? English isn't my first language.
user avatar
1 vote
1 answer
69 views

Compare only files within the same source folder vs. Compare only files within the same source and sub folder

Isn't 2 and 4 identical? Can you please distinguish them? Subfolders must be within the source folder. Then "within the same source folder" in 2 must include all subfolders. Compare files ...
user avatar
0 votes
0 answers
217 views

Anti-Twin's Byte by Byte" vs. "Compare images (pixels)"

Kindly see the red rectangle below. How does "Byte by Byte" differ from "Compare images (pixels)"? Which is more accurate? What are the pros and cons of each? Advise me if I got ...
user avatar
0 votes
1 answer
537 views

Best way to find duplicate data files between two Linux machines?

I have two Debian boxes at home that have various photos, audio, video and other data files on them. They are both disorganised, I'd like to organise them and remove duplicate files between them. ...
Murgatroyd's user avatar
5 votes
3 answers
7k views

Digikam (KDE) - How to remove all duplicate photos, but keep the reference images?

Though Digikam provides a nice feature of finding duplicate photos, it does not, for what I can see, make it easy to delete duplicates and keep original photos. I hope that the function is somewhere ...
Andrew's user avatar
  • 51
3 votes
3 answers
3k views

ZFS dedup performance optimization

I'm running a VDI server. Many of the VDI users are installing the same apps on their desktops, so zfs set dedup=on vdipool/myDesktop comes in handy. How can I optimize the performance impact of zfs ...
AllBlooming's user avatar
0 votes
0 answers
23 views

Why are the contents of my Downloads folder duplicating to my account folder?

I just went to my C:/Users/[username] folder and noticed that every file that is in my downloads folder is also in my [username] folder. Why is that happening and how can I get this to stop?
Eluchel's user avatar
  • 101
0 votes
1 answer
110 views

Looking for a way to remove duplicate entries with variable information

I am trying to remove duplicates having variable information. Column F = delete time Column Q = recipient code Column R = recipient address I need to remove duplicates that match on row ...
Jessie's user avatar
  • 1
1 vote
0 answers
184 views

How to deduplicate Virtualbox VMs

I'm generally creating a new clone of my existing VM with "linked clone" option to save space. However, for some reasons (backup/restore etc.), I end up with fully cloned machines thus using huge ...
ceremcem's user avatar
  • 635
2 votes
1 answer
6k views

How to find duplicate id entries in a xml file

How can I quickly narrow down duplicate id entries in a xml file, example: <entry id="A">... <entry id="B">... <entry id="A">... and output them id="A" dup 2 times Just to let ...
Lowestoflowest's user avatar
0 votes
1 answer
93 views

Removing duplicates

i've got plenty of those duplicates: 2017-11-22_3083_smokey_brights_-_come_to_terms.mp3 2017-12-01-Smokey_Brights_-_Come_To_Terms.mp3 How can I keep only one of each ? Unfortunately, their md5sum ...
user1109729's user avatar
1 vote
1 answer
537 views

Remove email (.eml) duplicates

I have a folder with around 50.000 emails in .eml format. There are many duplicates, even triplets or quadriplets of them, i suppose around 30.000 in total. I have tried to remove duplicates using ...
Ljubisa Livac's user avatar
1 vote
0 answers
450 views

Detect Near-Duplicate Audio Files [closed]

Suppose I have a few directories of somewhere around 4000 audio files, about half of which are FLAC and the other half are MP3. Suppose I want to detect, not by tag metadata, but only by audio data, ...
EllipticalInitial's user avatar
0 votes
0 answers
75 views

How to remove duplicate in batch by calling another script / utility / program

i'm looking for a way to remove duplicate pictures via a batch file, but after asking on stack overflow, it turned out not to be doable without using external utility or software. So i'm asking here, ...
Mattlau04's user avatar
1 vote
1 answer
5k views

How to safely consolidate duplicate folders on Google Drive and then delete the emptied ones, syncing the kept ones?

I was reinstalling my system, and I carelessly copied folders to the folder I use for Google Drive and started syncing at the same time. That resulted in my files being split between duplicate ...
Stonecraft's user avatar
0 votes
2 answers
41 views

De-duping a table based on an expression using one field

Mariadb: mysql Ver 15.1 Distrib 10.3.8-MariaDB I have a table built as a union of queries from several other tables. This table looks something like- +-------------+--------------+------+-----+-----...
Incans's user avatar
  • 31
1 vote
1 answer
1k views

Combine Excel rows with duplicate keywords, but unique values in other columns

I have an Excel sheet with keywords in column B. For each row there will be a value in one cell somewhere between column E and column CR. So, line 3 will have a keyword in column B be and a value in ...
Meghan Foote's user avatar
0 votes
1 answer
198 views

Multiple criteria unique values, and list

I have two issues that I am looking for help with. The first is to create a dynamic list of unique ID numbers for items matching "Horse" in column J on Sheet 2!, "Ball" in column I on Sheet 2!, and ...
Victoria G's user avatar
2 votes
0 answers
828 views

In-place delta writes from a pipe

I'm looking for a way to only overwrites the parts of a file that differ from what's supposed to be written to it, so that the file is changed with the least amount of writes to disk possible. My ...
Atemu's user avatar
  • 137
0 votes
1 answer
185 views

Script for deduplicating files and folders with a particular suffix

A botched OneDrive restoration has left me with many files and folders with a " (1)" or " (2)" suffix. I would like a script (Bash is fine as I have MinGW + Cygwin, or PowerShell) that would parse ...
cfp's user avatar
  • 283
1 vote
3 answers
2k views

How to merge duplicate folders with "name (1)", "name (1) (1)" etc. structure

Syncing between my Google Filestream, Google Drive, and Synology CloudSync and got all messed up and I'm left with hundreds of duplicate folders with the folder name followed by a "(1)" or "(2)" etc., ...
Josh's user avatar
  • 159
0 votes
2 answers
946 views

Is there a way to backup deduplicated NTFS drive with rsync or other linux tool?

Is there a Linux way or a tool that is aware of Windows Deduplication on NTFS? Here I use rsync to sync files from a AD file server shares. The shares are mounted, readonly, using mount.cifs on linux. ...
msmafra's user avatar
  • 271
2 votes
1 answer
93 views

Remove files in Directory B that also exist in Directory A

This page has tells how to remove files in Directory B that also exist in Directory A. I want to use the windows solution but without the copying. Can someone tell me how to edit the FOR line to ...
Steev43230's user avatar
0 votes
1 answer
452 views

rsync replace only matching files with hardlinks

How to replace files in dst/, that exactly matches by content with files from src/ with hardlinks to these (src/*) files? There is --hard-links option for rsync, which I want to use, but I don't want ...
Tomilov Anatoliy's user avatar