Questions tagged [deduplication]
Deduplication is the finding and removal of duplicate files, or to keep both files but share the same allocation unit on the storage medium
153
questions
0
votes
1
answer
49
views
In Linux with limited features (NAS box), how can I find duplicate files, then remove duplicates and substitute hardlinks to one file?
I have a NAS box running some version of Linux that I use for backing up anything and everything.
It is essentially an absolute certainty that some of the files are identical duplicates.
That being ...
- 121
0
votes
2
answers
200
views
Is there a way to select the first instance of a duplicated file in dupeguru?
dupeguru has helped me find duplicated files, but it only allows me to delete the second instance of duplicated files (as shown in the screenshot). I wonder if there's a way to delete the first ...
- 105
2
votes
1
answer
58
views
Can I control which file jdupes preserves when using hardlinks?
The jdupes deduplication program allows you to create hardlinks, so that the files are not deleted, but linked together so that the disk space is used only once for multiple copies.
Is there a way to ...
- 1,118
1
vote
1
answer
145
views
How to remove duplicated text on every line using Notepad++?
I want to remove duplicated words in each line using Notepad++.
Example:
Flooring Services, Carpet, Flooring Services, Tile, Flooring Services
In the above, Flooring Services is repeated 3x. I only ...
- 19
0
votes
1
answer
560
views
Removing duplicate rows in a CSV file as compared to another CSV file
I have a CSV file named CSV1 with over 5000 rows and 3 columns - last name, first name and email address. I have another file named CSV2 with about 2700 rows with the same 3 columns. I have to remove ...
0
votes
0
answers
95
views
ZFS deduplication table not paged out to RAM
I have a pool on which there is 3.41 TB of data, deduplication is enabled on the volumes, judging by the output of the
zpool status -D <pool_name>
...
dedup: DDT entries 73665285, size 696B on ...
- 1
0
votes
0
answers
118
views
Remove but 'remember' (to later restore) duplicate frames from video/ image sequence?
I'm upscaling animated content (via ChaiNNer) in which a number of frames (often 25-50%) are duplicates, wasting significant gpu time; I've been attempting to remove duplicates pre upscale but have ...
0
votes
1
answer
319
views
btrfs fi du: several sets of shared extents
I have a lot of files in various directories on a btrfs filesystem. They are all identical, and some may have been created using a "old-style" full copy, some are clones (cp --reflink). I ...
- 1,118
1
vote
0
answers
377
views
How to overwrite all files only if SAME size and exact duplicate while renaming if different size?
I'm using Teracopy, but Windows also does not have this function. Instead it wants me to replace the file only if DIFFERENT, or skip it if the name is the same. I want to do the opposite.
I need to ...
- 3,496
1
vote
2
answers
801
views
Can I find duplicate files with the same path in different locations?
I have two folders/directories: C:\MyData and C:\MyDataBackup and the person that owns those two folders/directories and does not remember if they have edited the files in the original or in the ...
- 157
11
votes
1
answer
631
views
In bash, how to find all copies of a given file in particular directories?
Let's say we have a file /a_long_path_1/foo.doc of size, say, 12345 bytes, and we would like to find all copies of this file in directories /a_long_path_2 and /a_long_path_3 including all their ...
user1757545
0
votes
0
answers
196
views
How can I look for duplicate files from different base folders in bash?
I know fdupes, which can find duplicate files from various directories I provide it:
fdupes -r <directoryA> <directoryB> <...>
But this seems to “throw all files into the same ...
- 1,487
1
vote
1
answer
1k
views
Remove duplicate files within specific directories, keeping them in another with Bash
I've seen a few questions on here regarding removing duplicate files in various circumstances but I can't find anything along the lines of what I'm trying to accomplish.
The closest I came was seeing ...
- 11
0
votes
2
answers
470
views
File Deduplication and Filtering
I had to run a TestDisk file recovery software on one of my drives. The result was a little over 16,000 files who's names are basically an alpha numeric string. Many of these files contain duplicate ...
- 103
0
votes
1
answer
207
views
Detect files with multiple extensions
I've a synology running a kind of linux OS.
I've transfered there a lot of photos.
Unfortunately, some are with .jpg and some with .JPG depending of the tool I used to import.
These are considered as ...
- 13
0
votes
0
answers
52
views
delete files from folder and subfolders named filename - copy
I have had an issue with Microsoft OneDrive and now I have many files named filename - copy.
I found This Post which says to use del "* - Copy.*" which works, within a given directory, but ...
- 11
0
votes
1
answer
922
views
Cleanup duplicate mails after fetchmail messed up the fetchids list
I use fetchmail since years, to get my mails from a free private email provider, always with the "keep" option. Now, due to an update in mailu, the fetchids file got messed up and I have ...
- 232
0
votes
0
answers
2k
views
Cannot install Deduplication role on Windows Server 2019 0x80073701
I want to install the Deduplication role on Windows Server 2019 but I always get an 0x80073701 Error. I tried all sfc and Dism, updates but there where no problems, I also reinstalled update ...
- 1
2
votes
2
answers
122
views
Find and delete near duplicate lines with certain punctuation
I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
- 35
-1
votes
5
answers
3k
views
Why do duplicate files have different checksums?
I am trying to remove hundreds of duplicate .PST files from a single folder containing archived .PST files from Outlook. I've tried a couple of Duplicate Remover apps, which rely on checksums to ...
3
votes
1
answer
223
views
Which duplicate should I remove?
A while ago I somehow managed to duplicate a lot of photos but didn't realise till many months later and now don't know which one is the duplicate. For the duplicates, there is a common pattern ...
- 33
1
vote
1
answer
1k
views
Removing Duplicate Files on MacOS Terminal Using MD5
I found the following command that makes use of md5 file hashes to find duplicate files and list them in a text file.
find ./ -type f -exec md5 {} \; | awk -F '=' '{print $2 "\t" $1}' | sort ...
- 13
0
votes
1
answer
2k
views
The Remove Duplicate Messages add on to Mozilla Thunderbird does not find all duplicates
I have upgraded my email hosting and copied bulk messages from folders in the old server to the same folder in the new server. However in some cases I have copied many messages multiple times so have ...
- 678
1
vote
1
answer
564
views
How to delete all duplicates of a file linked by CoW on BTRFS
The glory of Copy-on-Write (CoW) allows us to copy a file as many times as we like, without duplicating the used space.
I have used this functionality extensively (as well as manual deduplication of ...
- 111
1
vote
0
answers
132
views
Music managing software delete lower quality duplicates [closed]
Until the end of the last year I was one of the few people who used Google Play Music, and as it was discontinued, I downloaded all my music using Google Takeout. Now I have almost all of my songs in ...
- 1,885
0
votes
1
answer
1k
views
Explain like I'm 5 these 9 methods to compare similar pictures – Hash, MD5, SHA?
Please compare and contrast – like I'm 5, in SIMPLE English – all 9 methods below? English isn't my first language.
user1268013
1
vote
1
answer
69
views
Compare only files within the same source folder vs. Compare only files within the same source and sub folder
Isn't 2 and 4 identical? Can you please distinguish them? Subfolders must be within the source folder. Then "within the same source folder" in 2 must include all subfolders.
Compare files ...
user1268013
0
votes
0
answers
211
views
Anti-Twin's Byte by Byte" vs. "Compare images (pixels)"
Kindly see the red rectangle below. How does "Byte by Byte" differ from "Compare images (pixels)"? Which is more accurate? What are the pros and cons of each?
Advise me if I got ...
user1268013
0
votes
1
answer
519
views
Best way to find duplicate data files between two Linux machines?
I have two Debian boxes at home that have various photos, audio, video and other data files on them. They are both disorganised, I'd like to organise them and remove duplicate files between them. ...
- 26
5
votes
3
answers
7k
views
Digikam (KDE) - How to remove all duplicate photos, but keep the reference images?
Though Digikam provides a nice feature of finding duplicate photos, it does not, for what I can see, make it easy to delete duplicates and keep original photos. I hope that the function is somewhere ...
- 51
3
votes
3
answers
3k
views
ZFS dedup performance optimization
I'm running a VDI server. Many of the VDI users are installing the same apps on their desktops, so zfs set dedup=on vdipool/myDesktop comes in handy.
How can I optimize the performance impact of zfs ...
- 193
0
votes
0
answers
23
views
Why are the contents of my Downloads folder duplicating to my account folder?
I just went to my C:/Users/[username] folder and noticed that every file that is in my downloads folder is also in my [username] folder. Why is that happening and how can I get this to stop?
- 101
0
votes
1
answer
95
views
Looking for a way to remove duplicate entries with variable information
I am trying to remove duplicates having variable information.
Column F = delete time
Column Q = recipient code
Column R = recipient address
I need to remove duplicates that match on row ...
- 1
0
votes
0
answers
176
views
How to deduplicate Virtualbox VMs
I'm generally creating a new clone of my existing VM with "linked clone" option to save space. However, for some reasons (backup/restore etc.), I end up with fully cloned machines thus using huge ...
- 605
2
votes
1
answer
5k
views
How to find duplicate id entries in a xml file
How can I quickly narrow down duplicate id entries in a xml file, example:
<entry id="A">...
<entry id="B">...
<entry id="A">...
and output them
id="A" dup 2 times
Just to let ...
0
votes
1
answer
91
views
Removing duplicates
i've got plenty of those duplicates:
2017-11-22_3083_smokey_brights_-_come_to_terms.mp3
2017-12-01-Smokey_Brights_-_Come_To_Terms.mp3
How can I keep only one of each ?
Unfortunately, their md5sum ...
1
vote
1
answer
518
views
Remove email (.eml) duplicates
I have a folder with around 50.000 emails in .eml format. There are many duplicates, even triplets or quadriplets of them, i suppose around 30.000 in total.
I have tried to remove duplicates using ...
- 113
1
vote
0
answers
446
views
Detect Near-Duplicate Audio Files [closed]
Suppose I have a few directories of somewhere around 4000 audio files, about half of which are FLAC and the other half are MP3. Suppose I want to detect, not by tag metadata, but only by audio data, ...
- 1,560
0
votes
0
answers
70
views
How to remove duplicate in batch by calling another script / utility / program
i'm looking for a way to remove duplicate pictures via a batch file, but after asking on stack overflow, it turned out not to be doable without using external utility or software.
So i'm asking here, ...
1
vote
1
answer
5k
views
How to safely consolidate duplicate folders on Google Drive and then delete the emptied ones, syncing the kept ones?
I was reinstalling my system, and I carelessly copied folders to the folder I use for Google Drive and started syncing at the same time.
That resulted in my files being split between duplicate ...
- 328
0
votes
2
answers
40
views
De-duping a table based on an expression using one field
Mariadb: mysql Ver 15.1 Distrib 10.3.8-MariaDB
I have a table built as a union of queries from several other tables. This table looks something like-
+-------------+--------------+------+-----+-----...
- 31
1
vote
1
answer
1k
views
Combine Excel rows with duplicate keywords, but unique values in other columns
I have an Excel sheet with keywords in column B. For each row there will be a value in one cell somewhere between column E and column CR.
So, line 3 will have a keyword in column B be and a value in ...
- 11
0
votes
1
answer
191
views
Multiple criteria unique values, and list
I have two issues that I am looking for help with.
The first is to create a dynamic list of unique ID numbers for items matching "Horse" in column J on Sheet 2!, "Ball" in column I on Sheet 2!, and ...
2
votes
0
answers
825
views
In-place delta writes from a pipe
I'm looking for a way to only overwrites the parts of a file that differ from what's supposed to be written to it, so that the file is changed with the least amount of writes to disk possible.
My ...
- 137
0
votes
1
answer
183
views
Script for deduplicating files and folders with a particular suffix
A botched OneDrive restoration has left me with many files and folders with a " (1)" or " (2)" suffix.
I would like a script (Bash is fine as I have MinGW + Cygwin, or PowerShell) that would parse ...
- 283
1
vote
3
answers
2k
views
How to merge duplicate folders with "name (1)", "name (1) (1)" etc. structure
Syncing between my Google Filestream, Google Drive, and Synology CloudSync and got all messed up and I'm left with hundreds of duplicate folders with the folder name followed by a "(1)" or "(2)" etc., ...
- 159
0
votes
2
answers
932
views
Is there a way to backup deduplicated NTFS drive with rsync or other linux tool?
Is there a Linux way or a tool that is aware of Windows Deduplication on NTFS?
Here I use rsync to sync files from a AD file server shares.
The shares are mounted, readonly, using mount.cifs on linux.
...
- 271
2
votes
1
answer
91
views
Remove files in Directory B that also exist in Directory A
This page has tells how to remove files in Directory B that also exist in Directory A.
I want to use the windows solution but without the copying.
Can someone tell me how to edit the FOR line to ...
- 71
0
votes
1
answer
446
views
rsync replace only matching files with hardlinks
How to replace files in dst/, that exactly matches by content with files from src/ with hardlinks to these (src/*) files?
There is --hard-links option for rsync, which I want to use, but I don't want ...
- 261
0
votes
0
answers
3k
views
How to find "revision number" and appropriate Windows version to read files off an old deduped disk?
I have a disk that's come off a dead server.
It's clearly deduped - the classic errors 1305, 80070519 and "bad revision" when I try to copy the files show that. I've tried every Windows Server ...
- 1,695