Questions tagged [deduplication]
Deduplication is the finding and removal of duplicate files, or to keep both files but share the same allocation unit on the storage medium
149
questions
0
votes
1
answer
22
views
Removing duplicate rows in a CSV file as compared to another CSV file
I have a CSV file named CSV1 with over 5000 rows and 3 columns - last name, first name and email address. I have another file named CSV2 with about 2700 rows with the same 3 columns. I have to remove ...
0
votes
0
answers
42
views
ZFS deduplication table not paged out to RAM
I have a pool on which there is 3.41 TB of data, deduplication is enabled on the volumes, judging by the output of the
zpool status -D <pool_name>
...
dedup: DDT entries 73665285, size 696B on ...
- 1
0
votes
0
answers
33
views
Remove but 'remember' (to later restore) duplicate frames from video/ image sequence?
I'm upscaling animated content (via ChaiNNer) in which a number of frames (often 25-50%) are duplicates, wasting significant gpu time; I've been attempting to remove duplicates pre upscale but have ...
0
votes
1
answer
131
views
btrfs fi du: several sets of shared extents
I have a lot of files in various directories on a btrfs filesystem. They are all identical, and some may have been created using a "old-style" full copy, some are clones (cp --reflink). I ...
- 1,108
1
vote
0
answers
195
views
How to overwrite all files only if SAME size and exact duplicate while renaming if different size?
I'm using Teracopy, but Windows also does not have this function. Instead it wants me to replace the file only if DIFFERENT, or skip it if the name is the same. I want to do the opposite.
I need to ...
- 3,448
1
vote
1
answer
320
views
Can I find duplicate files with the same path in different locations?
I have two folders/directories: C:\MyData and C:\MyDataBackup and the person that owns those two folders/directories and does not remember if they have edited the files in the original or in the ...
- 157
11
votes
1
answer
596
views
In bash, how to find all copies of a given file in particular directories?
Let's say we have a file /a_long_path_1/foo.doc of size, say, 12345 bytes, and we would like to find all copies of this file in directories /a_long_path_2 and /a_long_path_3 including all their ...
user1757545
0
votes
0
answers
58
views
How can I look for duplicate files from different base folders in bash?
I know fdupes, which can find duplicate files from various directories I provide it:
fdupes -r <directoryA> <directoryB> <...>
But this seems to “throw all files into the same ...
- 1,407
1
vote
1
answer
524
views
Remove duplicate files within specific directories, keeping them in another with Bash
I've seen a few questions on here regarding removing duplicate files in various circumstances but I can't find anything along the lines of what I'm trying to accomplish.
The closest I came was seeing ...
- 11
0
votes
2
answers
237
views
File Deduplication and Filtering
I had to run a TestDisk file recovery software on one of my drives. The result was a little over 16,000 files who's names are basically an alpha numeric string. Many of these files contain duplicate ...
- 103
0
votes
1
answer
165
views
Detect files with multiple extensions
I've a synology running a kind of linux OS.
I've transfered there a lot of photos.
Unfortunately, some are with .jpg and some with .JPG depending of the tool I used to import.
These are considered as ...
- 13
0
votes
0
answers
37
views
delete files from folder and subfolders named filename - copy
I have had an issue with Microsoft OneDrive and now I have many files named filename - copy.
I found This Post which says to use del "* - Copy.*" which works, within a given directory, but ...
- 11
0
votes
1
answer
717
views
Cleanup duplicate mails after fetchmail messed up the fetchids list
I use fetchmail since years, to get my mails from a free private email provider, always with the "keep" option. Now, due to an update in mailu, the fetchids file got messed up and I have ...
- 222
0
votes
0
answers
2k
views
Cannot install Deduplication role on Windows Server 2019 0x80073701
I want to install the Deduplication role on Windows Server 2019 but I always get an 0x80073701 Error. I tried all sfc and Dism, updates but there where no problems, I also reinstalled update ...
- 1
2
votes
2
answers
105
views
Find and delete near duplicate lines with certain punctuation
I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
- 35
1
vote
3
answers
2k
views
Why do duplicate files have different checksums?
I am trying to remove hundreds of duplicate .PST files from a single folder containing archived .PST files from Outlook. I've tried a couple of Duplicate Remover apps, which rely on checksums to ...
3
votes
1
answer
163
views
Which duplicate should I remove?
A while ago I somehow managed to duplicate a lot of photos but didn't realise till many months later and now don't know which one is the duplicate. For the duplicates, there is a common pattern ...
- 33
1
vote
1
answer
793
views
Removing Duplicate Files on MacOS Terminal Using MD5
I found the following command that makes use of md5 file hashes to find duplicate files and list them in a text file.
find ./ -type f -exec md5 {} \; | awk -F '=' '{print $2 "\t" $1}' | sort ...
- 13
0
votes
1
answer
1k
views
The Remove Duplicate Messages add on to Mozilla Thunderbird does not find all duplicates
I have upgraded my email hosting and copied bulk messages from folders in the old server to the same folder in the new server. However in some cases I have copied many messages multiple times so have ...
- 578
0
votes
1
answer
380
views
How to delete all duplicates of a file linked by CoW on BTRFS
The glory of Copy-on-Write (CoW) allows us to copy a file as many times as we like, without duplicating the used space.
I have used this functionality extensively (as well as manual deduplication of ...
- 101
1
vote
0
answers
113
views
Music managing software delete lower quality duplicates [closed]
Until the end of the last year I was one of the few people who used Google Play Music, and as it was discontinued, I downloaded all my music using Google Takeout. Now I have almost all of my songs in ...
- 1,881
-1
votes
1
answer
808
views
Explain like I'm 5 these 9 methods to compare similar pictures – Hash, MD5, SHA?
Please compare and contrast – like I'm 5, in SIMPLE English – all 9 methods below? English isn't my first language.
user1268013
1
vote
1
answer
68
views
Compare only files within the same source folder vs. Compare only files within the same source and sub folder
Isn't 2 and 4 identical? Can you please distinguish them? Subfolders must be within the source folder. Then "within the same source folder" in 2 must include all subfolders.
Compare files ...
user1268013
0
votes
0
answers
165
views
Anti-Twin's Byte by Byte" vs. "Compare images (pixels)"
Kindly see the red rectangle below. How does "Byte by Byte" differ from "Compare images (pixels)"? Which is more accurate? What are the pros and cons of each?
Advise me if I got ...
user1268013
0
votes
1
answer
389
views
Best way to find duplicate data files between two Linux machines?
I have two Debian boxes at home that have various photos, audio, video and other data files on them. They are both disorganised, I'd like to organise them and remove duplicate files between them. ...
- 26
5
votes
3
answers
5k
views
Digikam (KDE) - How to remove all duplicate photos, but keep the reference images?
Though Digikam provides a nice feature of finding duplicate photos, it does not, for what I can see, make it easy to delete duplicates and keep original photos. I hope that the function is somewhere ...
- 51
3
votes
3
answers
3k
views
ZFS dedup performance optimization
I'm running a VDI server. Many of the VDI users are installing the same apps on their desktops, so zfs set dedup=on vdipool/myDesktop comes in handy.
How can I optimize the performance impact of zfs ...
- 193
0
votes
0
answers
19
views
Why are the contents of my Downloads folder duplicating to my account folder?
I just went to my C:/Users/[username] folder and noticed that every file that is in my downloads folder is also in my [username] folder. Why is that happening and how can I get this to stop?
- 101
0
votes
1
answer
81
views
Looking for a way to remove duplicate entries with variable information
I am trying to remove duplicates having variable information.
Column F = delete time
Column Q = recipient code
Column R = recipient address
I need to remove duplicates that match on row ...
- 1
0
votes
0
answers
155
views
How to deduplicate Virtualbox VMs
I'm generally creating a new clone of my existing VM with "linked clone" option to save space. However, for some reasons (backup/restore etc.), I end up with fully cloned machines thus using huge ...
- 585
2
votes
1
answer
4k
views
How to find duplicate id entries in a xml file
How can I quickly narrow down duplicate id entries in a xml file, example:
<entry id="A">...
<entry id="B">...
<entry id="A">...
and output them
id="A" dup 2 times
Just to let ...
0
votes
1
answer
81
views
Removing duplicates
i've got plenty of those duplicates:
2017-11-22_3083_smokey_brights_-_come_to_terms.mp3
2017-12-01-Smokey_Brights_-_Come_To_Terms.mp3
How can I keep only one of each ?
Unfortunately, their md5sum ...
1
vote
1
answer
481
views
Remove email (.eml) duplicates
I have a folder with around 50.000 emails in .eml format. There are many duplicates, even triplets or quadriplets of them, i suppose around 30.000 in total.
I have tried to remove duplicates using ...
- 113
1
vote
0
answers
360
views
Detect Near-Duplicate Audio Files [closed]
Suppose I have a few directories of somewhere around 4000 audio files, about half of which are FLAC and the other half are MP3. Suppose I want to detect, not by tag metadata, but only by audio data, ...
- 1,446
0
votes
0
answers
38
views
How to remove duplicate in batch by calling another script / utility / program
i'm looking for a way to remove duplicate pictures via a batch file, but after asking on stack overflow, it turned out not to be doable without using external utility or software.
So i'm asking here, ...
1
vote
1
answer
4k
views
How to safely consolidate duplicate folders on Google Drive and then delete the emptied ones, syncing the kept ones?
I was reinstalling my system, and I carelessly copied folders to the folder I use for Google Drive and started syncing at the same time.
That resulted in my files being split between duplicate ...
- 298
0
votes
2
answers
37
views
De-duping a table based on an expression using one field
Mariadb: mysql Ver 15.1 Distrib 10.3.8-MariaDB
I have a table built as a union of queries from several other tables. This table looks something like-
+-------------+--------------+------+-----+-----...
- 31
1
vote
1
answer
1k
views
Combine Excel rows with duplicate keywords, but unique values in other columns
I have an Excel sheet with keywords in column B. For each row there will be a value in one cell somewhere between column E and column CR.
So, line 3 will have a keyword in column B be and a value in ...
- 11
0
votes
1
answer
161
views
Multiple criteria unique values, and list
I have two issues that I am looking for help with.
The first is to create a dynamic list of unique ID numbers for items matching "Horse" in column J on Sheet 2!, "Ball" in column I on Sheet 2!, and ...
2
votes
0
answers
811
views
In-place delta writes from a pipe
I'm looking for a way to only overwrites the parts of a file that differ from what's supposed to be written to it, so that the file is changed with the least amount of writes to disk possible.
My ...
- 137
0
votes
1
answer
171
views
Script for deduplicating files and folders with a particular suffix
A botched OneDrive restoration has left me with many files and folders with a " (1)" or " (2)" suffix.
I would like a script (Bash is fine as I have MinGW + Cygwin, or PowerShell) that would parse ...
- 273
1
vote
3
answers
2k
views
How to merge duplicate folders with "name (1)", "name (1) (1)" etc. structure
Syncing between my Google Filestream, Google Drive, and Synology CloudSync and got all messed up and I'm left with hundreds of duplicate folders with the folder name followed by a "(1)" or "(2)" etc., ...
- 159
0
votes
2
answers
827
views
Is there a way to backup deduplicated NTFS drive with rsync or other linux tool?
Is there a Linux way or a tool that is aware of Windows Deduplication on NTFS?
Here I use rsync to sync files from a AD file server shares.
The shares are mounted, readonly, using mount.cifs on linux.
...
- 271
2
votes
1
answer
81
views
Remove files in Directory B that also exist in Directory A
This page has tells how to remove files in Directory B that also exist in Directory A.
I want to use the windows solution but without the copying.
Can someone tell me how to edit the FOR line to ...
- 71
0
votes
1
answer
401
views
rsync replace only matching files with hardlinks
How to replace files in dst/, that exactly matches by content with files from src/ with hardlinks to these (src/*) files?
There is --hard-links option for rsync, which I want to use, but I don't want ...
- 261
0
votes
0
answers
3k
views
How to find "revision number" and appropriate Windows version to read files off an old deduped disk?
I have a disk that's come off a dead server.
It's clearly deduped - the classic errors 1305, 80070519 and "bad revision" when I try to copy the files show that. I've tried every Windows Server ...
- 1,655
1
vote
1
answer
499
views
Find sub-folders containing only duplicate files
I am looking for a method (not including paid software) for finding all folders that contain only files with that are also in at least one other sub-folder of the parent directory. If used on a music ...
- 73
1
vote
1
answer
1k
views
(Win Server 2012 R2) Can't enable data deduplication
Windows Server 2012 R2 (Datacentre edition)
I cannot enable Data Deduplication. This particular volume previously had data deduplication enabled. Computer was changed from BIOS to UEFI boot, so ...
- 121
1
vote
1
answer
7k
views
Delete duplicate images with different file names
I have lots of images that have all the exact same metadata (size, the picture itself, etc) and the only difference between them is their file names.
So say I have 10 images that all look the same, I ...
- 611
0
votes
1
answer
109
views
How to find files that are duplicates of files in a certain folder?
I need to remove the duplicates in my computer, since i have around 5 copies of every single photo, in different folders. I want to delete the photos that also appear in a certain folder, in witch i ...
- 11