Questions tagged [deduplication]

Deduplication is the finding and removal of duplicate files.

Filter by
Sorted by
Tagged with
0
votes
1answer
15 views

The Remove Duplicate Messages add on to Mozilla Thunderbird does not find all duplicates

I have upgraded my email hosting and copied bulk messages from folders in the old server to the same folder in the new server. However in some cases I have copied many messages multiple times so have ...
0
votes
0answers
65 views

“Replace or Skip Files” is broken with Windows 10

Super strange behavior from newest version of Windows 10. There are 200+ file conflict when copying, but in the list which files do you want to keep? It is blank and white after 20 items. https://...
0
votes
1answer
25 views

How to delete all duplicates of a file linked by CoW on BTRFS

The glory of Copy-on-Write (CoW) allows us to copy a file as many times as we like, without duplicating the used space. I have used this functionality extensively (as well as manual deduplication of ...
0
votes
0answers
15 views

How to specify logical size when mkfs.btrfs

I'm setting up a btrfs filesystem where there are 10TB files with an average file size of 1GB - all videos. Normally I'd go for the maximum allowed logical size but I couldn't find out how to specify ...
1
vote
0answers
42 views

Music managing software delete lower quality duplicates [closed]

Until the end of the last year I was one of the few people who used Google Play Music, and as it was discontinued, I downloaded all my music using Google Takeout. Now I have almost all of my songs in ...
-1
votes
1answer
63 views

Explain like I'm 5 these 9 methods to compare similar pictures – Hash, MD5, SHA?

Please compare and contrast – like I'm 5, in SIMPLE English – all 9 methods below? English isn't my first language.
1
vote
1answer
42 views

Compare only files within the same source folder vs. Compare only files within the same source and sub folder

Isn't 2 and 4 identical? Can you please distinguish them? Subfolders must be within the source folder. Then "within the same source folder" in 2 must include all subfolders. Compare files ...
0
votes
0answers
43 views

Anti-Twin's Byte by Byte“ vs. ”Compare images (pixels)"

Kindly see the red rectangle below. How does "Byte by Byte" differ from "Compare images (pixels)"? Which is more accurate? What are the pros and cons of each? Advise me if I got ...
0
votes
1answer
46 views

Best way to find duplicate data files between two Linux machines?

I have two Debian boxes at home that have various photos, audio, video and other data files on them. They are both disorganised, I'd like to organise them and remove duplicate files between them. ...
0
votes
0answers
83 views

Shrink ZFS deduplication table (DDT), disable dedup for good

I could not resist the temptation of giving ZFS deduplication a try, and now I want to get rid of the daemons of the past. Or in more technical terms, I have this... dedup: DDT entries 1481316, size ...
2
votes
1answer
2k views

Digikam (KDE) - How to remove all duplicate photos, but keep the reference images?

Though Digikam provides a nice feature of finding duplicate photos, it does not, for what I can see, make it easy to delete duplicates and keep original photos. I hope that the function is somewhere ...
3
votes
3answers
665 views

ZFS dedup performance optimization

I'm running a VDI server. Many of the VDI users are installing the same apps on their desktops, so zfs set dedup=on vdipool/myDesktop comes in handy. How can I optimize the performance impact of zfs ...
0
votes
0answers
16 views

Why are the contents of my Downloads folder duplicating to my account folder?

I just went to my C:/Users/[username] folder and noticed that every file that is in my downloads folder is also in my [username] folder. Why is that happening and how can I get this to stop?
0
votes
1answer
71 views

Looking for a way to remove duplicate entries with variable information

I am trying to remove duplicates having variable information. Column F = delete time Column Q = recipient code Column R = recipient address I need to remove duplicates that match on row ...
0
votes
0answers
67 views

How to deduplicate Virtualbox VMs

I'm generally creating a new clone of my existing VM with "linked clone" option to save space. However, for some reasons (backup/restore etc.), I end up with fully cloned machines thus using huge ...
1
vote
1answer
1k views

How to find duplicate id entries in a xml file

How can I quickly narrow down duplicate id entries in a xml file, example: <entry id="A">... <entry id="B">... <entry id="A">... and output them id="A" dup 2 times Just to let ...
0
votes
1answer
65 views

Removing duplicates

i've got plenty of those duplicates: 2017-11-22_3083_smokey_brights_-_come_to_terms.mp3 2017-12-01-Smokey_Brights_-_Come_To_Terms.mp3 How can I keep only one of each ? Unfortunately, their md5sum ...
0
votes
1answer
243 views

Remove email (.eml) duplicates

I have a folder with around 50.000 emails in .eml format. There are many duplicates, even triplets or quadriplets of them, i suppose around 30.000 in total. I have tried to remove duplicates using ...
2
votes
0answers
120 views

Detect Near-Duplicate Audio Files [closed]

Suppose I have a few directories of somewhere around 4000 audio files, about half of which are FLAC and the other half are MP3. Suppose I want to detect, not by tag metadata, but only by audio data, ...
0
votes
0answers
24 views

How to remove duplicate in batch by calling another script / utility / program

i'm looking for a way to remove duplicate pictures via a batch file, but after asking on stack overflow, it turned out not to be doable without using external utility or software. So i'm asking here, ...
1
vote
1answer
1k views

How to safely consolidate duplicate folders on Google Drive and then delete the emptied ones, syncing the kept ones?

I was reinstalling my system, and I carelessly copied folders to the folder I use for Google Drive and started syncing at the same time. That resulted in my files being split between duplicate ...
0
votes
2answers
32 views

De-duping a table based on an expression using one field

Mariadb: mysql Ver 15.1 Distrib 10.3.8-MariaDB I have a table built as a union of queries from several other tables. This table looks something like- +-------------+--------------+------+-----+-----...
1
vote
1answer
984 views

Combine Excel rows with duplicate keywords, but unique values in other columns

I have an Excel sheet with keywords in column B. For each row there will be a value in one cell somewhere between column E and column CR. So, line 3 will have a keyword in column B be and a value in ...
0
votes
1answer
95 views

Multiple criteria unique values, and list

I have two issues that I am looking for help with. The first is to create a dynamic list of unique ID numbers for items matching "Horse" in column J on Sheet 2!, "Ball" in column I on Sheet 2!, and ...
2
votes
0answers
758 views

In-place delta writes from a pipe

I'm looking for a way to only overwrites the parts of a file that differ from what's supposed to be written to it, so that the file is changed with the least amount of writes to disk possible. My ...
0
votes
1answer
122 views

Script for deduplicating files and folders with a particular suffix

A botched OneDrive restoration has left me with many files and folders with a " (1)" or " (2)" suffix. I would like a script (Bash is fine as I have MinGW + Cygwin, or PowerShell) that would parse ...
1
vote
3answers
1k views

How to merge duplicate folders with “name (1)”, “name (1) (1)” etc. structure

Syncing between my Google Filestream, Google Drive, and Synology CloudSync and got all messed up and I'm left with hundreds of duplicate folders with the folder name followed by a "(1)" or "(2)" etc., ...
0
votes
2answers
532 views

Is there a way to backup deduplicated NTFS drive with rsync or other linux tool?

Is there a Linux way or a tool that is aware of Windows Deduplication on NTFS? Here I use rsync to sync files from a AD file server shares. The shares are mounted, readonly, using mount.cifs on linux. ...
2
votes
1answer
68 views

Remove files in Directory B that also exist in Directory A

This page has tells how to remove files in Directory B that also exist in Directory A. I want to use the windows solution but without the copying. Can someone tell me how to edit the FOR line to ...
0
votes
1answer
252 views

rsync replace only matching files with hardlinks

How to replace files in dst/, that exactly matches by content with files from src/ with hardlinks to these (src/*) files? There is --hard-links option for rsync, which I want to use, but I don't want ...
0
votes
0answers
3k views

How to find “revision number” and appropriate Windows version to read files off an old deduped disk?

I have a disk that's come off a dead server. It's clearly deduped - the classic errors 1305, 80070519 and "bad revision" when I try to copy the files show that. I've tried every Windows Server ...
1
vote
1answer
195 views

Find sub-folders containing only duplicate files

I am looking for a method (not including paid software) for finding all folders that contain only files with that are also in at least one other sub-folder of the parent directory. If used on a music ...
1
vote
1answer
772 views

(Win Server 2012 R2) Can't enable data deduplication

Windows Server 2012 R2 (Datacentre edition) I cannot enable Data Deduplication. This particular volume previously had data deduplication enabled. Computer was changed from BIOS to UEFI boot, so ...
1
vote
1answer
5k views

Delete duplicate images with different file names

I have lots of images that have all the exact same metadata (size, the picture itself, etc) and the only difference between them is their file names. So say I have 10 images that all look the same, I ...
0
votes
1answer
99 views

How to find files that are duplicates of files in a certain folder?

I need to remove the duplicates in my computer, since i have around 5 copies of every single photo, in different folders. I want to delete the photos that also appear in a certain folder, in witch i ...
2
votes
1answer
2k views

How Borg backup handle data deduplication and ecryption?

I need a backup software which performs encryption clientside. On Borg Documentation it is stated that it performs the encryption clientside and eventually if the repository is remote, another ...
1
vote
2answers
984 views

Find duplicate hashes

I have a bunch of files (4800 files), that I believe contain many duplicates and triples. Problem is that they don't also have the same names. I already weeded out the easy ones with same names. The ...
0
votes
1answer
879 views

How do I exclude/ignore specific file types/extensions with rmlint?

I am trying to dedupe a labyrinthine directory, but I keep getting results for the contents of the items I want to match rather than the enclosing items, themselves, i.e. my results are full of things ...
3
votes
3answers
2k views

Find and delete duplicated files in different disks and directories

I have hundreds of thousands files distributed in many external disks and disks in computers and many are duplicated. This mess was caused by myself creating copies for safety proposal. From time to ...
6
votes
1answer
6k views

zfs on linux compression and dedup order

What is the order for data written to a zfs filesystem on zfs on linux? The only specific document i found at http://docs.oracle.com/cd/E36784_01/html/E36835/gkknx.html says; When a file is written, ...
0
votes
1answer
58 views

Removing Duplicates From 1 Column Assocatied with a Name in Multiple Rows [duplicate]

I'm trying to create a pivot table to count the number of events a person has attended. In the example below, I'd like to remove the duplicates associated with a specific person. Joyce is listed ...
0
votes
0answers
105 views

Removing replies in downloaded emails (i.e., non-consecutive duplicate sentences from txt file)

Hello Experts and Fellow Learners! I have dozens of text files that contain monthly archives of a listserv. I'm trying to eliminate/delete quoted replies so that each message only appears once across ...
0
votes
1answer
2k views

(Win Server 2012) Can't enable Data Deduplication on C: drive

I'm trying to enable Data Deduplication on C: drive but Server Manager doesn't allow me to do so. After using PowerShell command it gave me following message: Windows PowerShell Copyright (C) 2014 ...
0
votes
1answer
300 views

I have a situation where data-deduplication in Server 2012 has corrupted the metadata - how can I copy these files from an old repository?

I have two drives - drive D: and drive F:. Both of these are drives locally attached to the Server 2012 machine. drive F is a legacy store of data migrating to a new drive (D) as it's larger. Data ...
1
vote
1answer
3k views

Batch to sort a file with removing duplicates line?

If I have a text file, I can sort it by command sort input.txt output.txt. Can I somehow also remove duplicate lines from the text file as well. (Ideally I would like a solution which can be done in a ...
2
votes
1answer
3k views

How to reduce memory usage after setting dedup=off (after having it on for a while)?

I know that setting dedup=on causes high memory usage and that it's almost never worth it (see e.g. https://serverfault.com/a/569099/208661). However I did turn it on in order to see how it works and ...
20
votes
5answers
18k views

Is there a way to extract duplicate lines in Sublime Text?

I need to perform 2 operations in Sublime Text: extract unique lines and extract duplicate lines. For example for input a b a Extract duplicates should result in: a and Extract unique should ...
2
votes
2answers
381 views

Optimizing gzipped files for block-level deduplication

I have about 100TiB of compressed gzip data on cloud volumes. On my way to run a block-level dedup (duperemove for btrfs) I find that it was compressed without -n, which causes block-level difference ...
0
votes
1answer
82 views

How to find extracted Zip files and their folder in mac

How can i list all zip files and their extracted folders in order to remove one of them, i downloaded lots of zip files in years(like everybody) and extracted them so now they are taking twice as ...
1
vote
0answers
355 views

How to batch delete files based on their checksum matching an existing list of checksums

I'm looking for a means of deleting files based on their checksum matching a list of previously seen checksums in a text file or some other list. The context is a large photo organization project ...