Questions tagged [deduplication]

Deduplication is the finding and removal of duplicate files, or to keep both files but share the same allocation unit on the storage medium

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

delete files from folder and subfolders named filename - copy

I have had an issue with Microsoft OneDrive and now I have many files named filename - copy. I found This Post which says to use del "* - Copy.*" which works, within a given directory, but ...
user avatar
  • 11
0 votes
1 answer
56 views

Cleanup duplicate mails after fetchmail messed up the fetchids list

I use fetchmail since years, to get my mails from a free private email provider, always with the "keep" option. Now, due to an update in mailu, the fetchids file got messed up and I have ...
user avatar
0 votes
0 answers
375 views

Cannot install Deduplication role on Windows Server 2019 0x80073701

I want to install the Deduplication role on Windows Server 2019 but I always get an 0x80073701 Error. I tried all sfc and Dism, updates but there where no problems, I also reinstalled update ...
user avatar
  • 1
0 votes
0 answers
28 views

mounted a btrfs volume with nodatasum for performance… How to write the checksums for the files created that way?

Using a performance critical database being several Tb large, I mounted a btrfs volume using the nodatasum mount option. As a result, more than 50% of the space is used by newly created files without ...
user avatar
  • 1,681
2 votes
2 answers
60 views

Find and delete near duplicate lines with certain punctuation

I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
user avatar
  • 35
0 votes
2 answers
201 views

Why do duplicate files have different checksums?

I am trying to remove hundreds of duplicate .PST files from a single folder containing archived .PST files from Outlook. I've tried a couple of Duplicate Remover apps, which rely on checksums to ...
user avatar
3 votes
1 answer
60 views

Which duplicate should I remove?

A while ago I somehow managed to duplicate a lot of photos but didn't realise till many months later and now don't know which one is the duplicate. For the duplicates, there is a common pattern ...
user avatar
0 votes
0 answers
20 views

Identifying entire duplicated rows

I have a data sheet with 15 columns and 16,000 rows. I need to identify which entire rows are the exact same, not just rows with certain cells that are duplicates. When I try to apply conditional ...
user avatar
0 votes
0 answers
142 views

How does gzip --rsyncable work from a technical perspective?

I know that gzip --rsyncable compresses files that way that different compressed versions of the same file differ less so rsycs delta-transfer can transfer them with less traffic https://linuxcommand....
user avatar
  • 119
1 vote
1 answer
214 views

Removing Duplicate Files on MacOS Terminal Using MD5

I found the following command that makes use of md5 file hashes to find duplicate files and list them in a text file. find ./ -type f -exec md5 {} \; | awk -F '=' '{print $2 "\t" $1}' | sort ...
user avatar
  • 13
0 votes
1 answer
332 views

The Remove Duplicate Messages add on to Mozilla Thunderbird does not find all duplicates

I have upgraded my email hosting and copied bulk messages from folders in the old server to the same folder in the new server. However in some cases I have copied many messages multiple times so have ...
user avatar
  • 508
0 votes
1 answer
107 views

How to delete all duplicates of a file linked by CoW on BTRFS

The glory of Copy-on-Write (CoW) allows us to copy a file as many times as we like, without duplicating the used space. I have used this functionality extensively (as well as manual deduplication of ...
user avatar
  • 101
1 vote
0 answers
61 views

Music managing software delete lower quality duplicates [closed]

Until the end of the last year I was one of the few people who used Google Play Music, and as it was discontinued, I downloaded all my music using Google Takeout. Now I have almost all of my songs in ...
user avatar
  • 1,497
-1 votes
1 answer
202 views

Explain like I'm 5 these 9 methods to compare similar pictures – Hash, MD5, SHA?

Please compare and contrast – like I'm 5, in SIMPLE English – all 9 methods below? English isn't my first language.
user avatar
1 vote
1 answer
49 views

Compare only files within the same source folder vs. Compare only files within the same source and sub folder

Isn't 2 and 4 identical? Can you please distinguish them? Subfolders must be within the source folder. Then "within the same source folder" in 2 must include all subfolders. Compare files ...
user avatar
0 votes
0 answers
93 views

Anti-Twin's Byte by Byte" vs. "Compare images (pixels)"

Kindly see the red rectangle below. How does "Byte by Byte" differ from "Compare images (pixels)"? Which is more accurate? What are the pros and cons of each? Advise me if I got ...
user avatar
0 votes
1 answer
75 views

Best way to find duplicate data files between two Linux machines?

I have two Debian boxes at home that have various photos, audio, video and other data files on them. They are both disorganised, I'd like to organise them and remove duplicate files between them. ...
user avatar
4 votes
1 answer
3k views

Digikam (KDE) - How to remove all duplicate photos, but keep the reference images?

Though Digikam provides a nice feature of finding duplicate photos, it does not, for what I can see, make it easy to delete duplicates and keep original photos. I hope that the function is somewhere ...
user avatar
  • 41
3 votes
3 answers
2k views

ZFS dedup performance optimization

I'm running a VDI server. Many of the VDI users are installing the same apps on their desktops, so zfs set dedup=on vdipool/myDesktop comes in handy. How can I optimize the performance impact of zfs ...
user avatar
0 votes
0 answers
18 views

Why are the contents of my Downloads folder duplicating to my account folder?

I just went to my C:/Users/[username] folder and noticed that every file that is in my downloads folder is also in my [username] folder. Why is that happening and how can I get this to stop?
user avatar
  • 101
0 votes
1 answer
76 views

Looking for a way to remove duplicate entries with variable information

I am trying to remove duplicates having variable information. Column F = delete time Column Q = recipient code Column R = recipient address I need to remove duplicates that match on row ...
user avatar
  • 1
0 votes
0 answers
103 views

How to deduplicate Virtualbox VMs

I'm generally creating a new clone of my existing VM with "linked clone" option to save space. However, for some reasons (backup/restore etc.), I end up with fully cloned machines thus using huge ...
user avatar
  • 495
1 vote
1 answer
2k views

How to find duplicate id entries in a xml file

How can I quickly narrow down duplicate id entries in a xml file, example: <entry id="A">... <entry id="B">... <entry id="A">... and output them id="A" dup 2 times Just to let ...
user avatar
0 votes
1 answer
69 views

Removing duplicates

i've got plenty of those duplicates: 2017-11-22_3083_smokey_brights_-_come_to_terms.mp3 2017-12-01-Smokey_Brights_-_Come_To_Terms.mp3 How can I keep only one of each ? Unfortunately, their md5sum ...
user avatar
0 votes
1 answer
327 views

Remove email (.eml) duplicates

I have a folder with around 50.000 emails in .eml format. There are many duplicates, even triplets or quadriplets of them, i suppose around 30.000 in total. I have tried to remove duplicates using ...
user avatar
2 votes
0 answers
188 views

Detect Near-Duplicate Audio Files [closed]

Suppose I have a few directories of somewhere around 4000 audio files, about half of which are FLAC and the other half are MP3. Suppose I want to detect, not by tag metadata, but only by audio data, ...
user avatar
  • 906
0 votes
0 answers
28 views

How to remove duplicate in batch by calling another script / utility / program

i'm looking for a way to remove duplicate pictures via a batch file, but after asking on stack overflow, it turned out not to be doable without using external utility or software. So i'm asking here, ...
user avatar
1 vote
1 answer
3k views

How to safely consolidate duplicate folders on Google Drive and then delete the emptied ones, syncing the kept ones?

I was reinstalling my system, and I carelessly copied folders to the folder I use for Google Drive and started syncing at the same time. That resulted in my files being split between duplicate ...
user avatar
0 votes
2 answers
35 views

De-duping a table based on an expression using one field

Mariadb: mysql Ver 15.1 Distrib 10.3.8-MariaDB I have a table built as a union of queries from several other tables. This table looks something like- +-------------+--------------+------+-----+-----...
user avatar
  • 31
1 vote
1 answer
1k views

Combine Excel rows with duplicate keywords, but unique values in other columns

I have an Excel sheet with keywords in column B. For each row there will be a value in one cell somewhere between column E and column CR. So, line 3 will have a keyword in column B be and a value in ...
user avatar
0 votes
1 answer
134 views

Multiple criteria unique values, and list

I have two issues that I am looking for help with. The first is to create a dynamic list of unique ID numbers for items matching "Horse" in column J on Sheet 2!, "Ball" in column I on Sheet 2!, and ...
user avatar
2 votes
0 answers
777 views

In-place delta writes from a pipe

I'm looking for a way to only overwrites the parts of a file that differ from what's supposed to be written to it, so that the file is changed with the least amount of writes to disk possible. My ...
user avatar
  • 139
0 votes
1 answer
141 views

Script for deduplicating files and folders with a particular suffix

A botched OneDrive restoration has left me with many files and folders with a " (1)" or " (2)" suffix. I would like a script (Bash is fine as I have MinGW + Cygwin, or PowerShell) that would parse ...
user avatar
  • 273
1 vote
3 answers
1k views

How to merge duplicate folders with "name (1)", "name (1) (1)" etc. structure

Syncing between my Google Filestream, Google Drive, and Synology CloudSync and got all messed up and I'm left with hundreds of duplicate folders with the folder name followed by a "(1)" or "(2)" etc., ...
user avatar
  • 159
0 votes
2 answers
639 views

Is there a way to backup deduplicated NTFS drive with rsync or other linux tool?

Is there a Linux way or a tool that is aware of Windows Deduplication on NTFS? Here I use rsync to sync files from a AD file server shares. The shares are mounted, readonly, using mount.cifs on linux. ...
user avatar
  • 261
2 votes
1 answer
73 views

Remove files in Directory B that also exist in Directory A

This page has tells how to remove files in Directory B that also exist in Directory A. I want to use the windows solution but without the copying. Can someone tell me how to edit the FOR line to ...
user avatar
0 votes
1 answer
303 views

rsync replace only matching files with hardlinks

How to replace files in dst/, that exactly matches by content with files from src/ with hardlinks to these (src/*) files? There is --hard-links option for rsync, which I want to use, but I don't want ...
user avatar
0 votes
0 answers
3k views

How to find "revision number" and appropriate Windows version to read files off an old deduped disk?

I have a disk that's come off a dead server. It's clearly deduped - the classic errors 1305, 80070519 and "bad revision" when I try to copy the files show that. I've tried every Windows Server ...
user avatar
  • 1,505
1 vote
1 answer
253 views

Find sub-folders containing only duplicate files

I am looking for a method (not including paid software) for finding all folders that contain only files with that are also in at least one other sub-folder of the parent directory. If used on a music ...
user avatar
1 vote
1 answer
934 views

(Win Server 2012 R2) Can't enable data deduplication

Windows Server 2012 R2 (Datacentre edition) I cannot enable Data Deduplication. This particular volume previously had data deduplication enabled. Computer was changed from BIOS to UEFI boot, so ...
user avatar
1 vote
1 answer
6k views

Delete duplicate images with different file names

I have lots of images that have all the exact same metadata (size, the picture itself, etc) and the only difference between them is their file names. So say I have 10 images that all look the same, I ...
user avatar
0 votes
1 answer
105 views

How to find files that are duplicates of files in a certain folder?

I need to remove the duplicates in my computer, since i have around 5 copies of every single photo, in different folders. I want to delete the photos that also appear in a certain folder, in witch i ...
user avatar
  • 11
2 votes
1 answer
2k views

How Borg backup handle data deduplication and ecryption?

I need a backup software which performs encryption clientside. On Borg Documentation it is stated that it performs the encryption clientside and eventually if the repository is remote, another ...
user avatar
  • 137
1 vote
2 answers
1k views

Find duplicate hashes

I have a bunch of files (4800 files), that I believe contain many duplicates and triples. Problem is that they don't also have the same names. I already weeded out the easy ones with same names. The ...
user avatar
  • 169
0 votes
1 answer
1k views

How do I exclude/ignore specific file types/extensions with rmlint?

I am trying to dedupe a labyrinthine directory, but I keep getting results for the contents of the items I want to match rather than the enclosing items, themselves, i.e. my results are full of things ...
user avatar
  • 101
3 votes
3 answers
2k views

Find and delete duplicated files in different disks and directories

I have hundreds of thousands files distributed in many external disks and disks in computers and many are duplicated. This mess was caused by myself creating copies for safety proposal. From time to ...
user avatar
  • 43
6 votes
1 answer
7k views

zfs on linux compression and dedup order

What is the order for data written to a zfs filesystem on zfs on linux? The only specific document i found at http://docs.oracle.com/cd/E36784_01/html/E36835/gkknx.html says; When a file is written, ...
user avatar
0 votes
1 answer
60 views

Removing Duplicates From 1 Column Assocatied with a Name in Multiple Rows [duplicate]

I'm trying to create a pivot table to count the number of events a person has attended. In the example below, I'd like to remove the duplicates associated with a specific person. Joyce is listed ...
user avatar
0 votes
0 answers
111 views

Removing replies in downloaded emails (i.e., non-consecutive duplicate sentences from txt file)

Hello Experts and Fellow Learners! I have dozens of text files that contain monthly archives of a listserv. I'm trying to eliminate/delete quoted replies so that each message only appears once across ...
user avatar
  • 1
0 votes
1 answer
3k views

(Win Server 2012) Can't enable Data Deduplication on C: drive

I'm trying to enable Data Deduplication on C: drive but Server Manager doesn't allow me to do so. After using PowerShell command it gave me following message: Windows PowerShell Copyright (C) 2014 ...
user avatar
  • 700