The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [deduplication]

Deduplication is the finding and removal of duplicate files.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Download from BidContender.com to Windows network drive creating endless duplicates

Problem that started last week. Zip file downloaded through Safari on Windows computer A to network drive on Windows computer B from bidcontender.com Network drive is a Sharepoint 365 sync Multiple 0 ...
0
votes
1answer
63 views

Remove email (.eml) duplicates

I have a folder with around 50.000 emails in .eml format. There are many duplicates, even triplets or quadriplets of them, i suppose around 30.000 in total. I have tried to remove duplicates using ...
2
votes
0answers
31 views

Detect Near-Duplicate Audio Files [closed]

Suppose I have a few directories of somewhere around 4000 audio files, about half of which are FLAC and the other half are MP3. Suppose I want to detect, not by tag metadata, but only by audio data, ...
0
votes
0answers
39 views

Many fragments of files recovered, how to find which ones belong to existing files

I recovered the contents of a reformatted 4TB HDD, using R-Studio, which analyses storage volumes both based on file-system informations, and by way of “raw file carving”, based on the detection of ...
0
votes
0answers
20 views

How to remove duplicate in batch by calling another script / utility / program

i'm looking for a way to remove duplicate pictures via a batch file, but after asking on stack overflow, it turned out not to be doable without using external utility or software. So i'm asking here, ...
0
votes
1answer
35 views

How to safely consolidate duplicate folders on Google Drive and then delete the emptied ones, syncing the kept ones?

I was reinstalling my system, and I carelessly copied folders to the folder I use for Google Drive and started syncing at the same time. That resulted in my files being split between duplicate ...
0
votes
0answers
20 views

How to check if there exist duplicates of a file in cloud while offline?

I have some files archived and uploaded to Google Drive, however I don't keep the data on my computer after they are uploaded. Frequently, I download a bunch of files that I'd like to add to the ...
0
votes
2answers
26 views

De-duping a table based on an expression using one field

Mariadb: mysql Ver 15.1 Distrib 10.3.8-MariaDB I have a table built as a union of queries from several other tables. This table looks something like- +-------------+--------------+------+-----+-----...
1
vote
1answer
118 views

Combine Excel rows with duplicate keywords, but unique values in other columns

I have an Excel sheet with keywords in column B. For each row there will be a value in one cell somewhere between column E and column CR. So, line 3 will have a keyword in column B be and a value in ...
0
votes
1answer
54 views

Multiple criteria unique values, and list

I have two issues that I am looking for help with. The first is to create a dynamic list of unique ID numbers for items matching "Horse" in column J on Sheet 2!, "Ball" in column I on Sheet 2!, and ...
2
votes
0answers
711 views

In-place delta writes from a pipe

I'm looking for a way to only overwrites the parts of a file that differ from what's supposed to be written to it, so that the file is changed with the least amount of writes to disk possible. My ...
0
votes
1answer
90 views

Script for deduplicating files and folders with a particular suffix

A botched OneDrive restoration has left me with many files and folders with a " (1)" or " (2)" suffix. I would like a script (Bash is fine as I have MinGW + Cygwin, or PowerShell) that would parse ...
1
vote
3answers
601 views

How to merge duplicate folders with “name (1)”, “name (1) (1)” etc. structure

Syncing between my Google Filestream, Google Drive, and Synology CloudSync and got all messed up and I'm left with hundreds of duplicate folders with the folder name followed by a "(1)" or "(2)" etc., ...
0
votes
2answers
306 views

Is there a way to backup deduplicated NTFS drive with rsync or other linux tool?

Is there a Linux way or a tool that is aware of Windows Deduplication on NTFS? Here I use rsync to sync files from a AD file server shares. The shares are mounted, readonly, using mount.cifs on linux. ...
1
vote
0answers
32 views

Remove files in Directory B that also exist in Directory A

This page has tells how to remove files in Directory B that also exist in Directory A. I want to use the windows solution but without the copying. Can someone tell me how to edit the FOR line to ...
0
votes
1answer
168 views

rsync replace only matching files with hardlinks

How to replace files in dst/, that exactly matches by content with files from src/ with hardlinks to these (src/*) files? There is --hard-links option for rsync, which I want to use, but I don't want ...
0
votes
0answers
3k views

How to find “revision number” and appropriate Windows version to read files off an old deduped disk?

I have a disk that's come off a dead server. It's clearly deduped - the classic errors 1305, 80070519 and "bad revision" when I try to copy the files show that. I've tried every Windows Server ...
1
vote
1answer
117 views

Find sub-folders containing only duplicate files

I am looking for a method (not including paid software) for finding all folders that contain only files with that are also in at least one other sub-folder of the parent directory. If used on a music ...
1
vote
1answer
413 views

(Win Server 2012 R2) Can't enable data deduplication

Windows Server 2012 R2 (Datacentre edition) I cannot enable Data Deduplication. This particular volume previously had data deduplication enabled. Computer was changed from BIOS to UEFI boot, so ...
1
vote
1answer
2k views

Delete duplicate images with different file names

I have lots of images that have all the exact same metadata (size, the picture itself, etc) and the only difference between them is their file names. So say I have 10 images that all look the same, I ...
0
votes
1answer
92 views

How to find files that are duplicates of files in a certain folder?

I need to remove the duplicates in my computer, since i have around 5 copies of every single photo, in different folders. I want to delete the photos that also appear in a certain folder, in witch i ...
1
vote
1answer
1k views

How Borg backup handle data deduplication and ecryption?

I need a backup software which performs encryption clientside. On Borg Documentation it is stated that it performs the encryption clientside and eventually if the repository is remote, another ...
1
vote
1answer
584 views

Find duplicate hashes

I have a bunch of files (4800 files), that I believe contain many duplicates and triples. Problem is that they don't also have the same names. I already weeded out the easy ones with same names. The ...
0
votes
1answer
555 views

How do I exclude/ignore specific file types/extensions with rmlint?

I am trying to dedupe a labyrinthine directory, but I keep getting results for the contents of the items I want to match rather than the enclosing items, themselves, i.e. my results are full of things ...
1
vote
3answers
1k views

Find and delete duplicated files in different disks and directories

I have hundreds of thousands files distributed in many external disks and disks in computers and many are duplicated. This mess was caused by myself creating copies for safety proposal. From time to ...
4
votes
1answer
4k views

zfs on linux compression and dedup order

What is the order for data written to a zfs filesystem on zfs on linux? The only specific document i found at http://docs.oracle.com/cd/E36784_01/html/E36835/gkknx.html says; When a file is written, ...
0
votes
1answer
56 views

Removing Duplicates From 1 Column Assocatied with a Name in Multiple Rows [duplicate]

I'm trying to create a pivot table to count the number of events a person has attended. In the example below, I'd like to remove the duplicates associated with a specific person. Joyce is listed ...
0
votes
0answers
92 views

Removing replies in downloaded emails (i.e., non-consecutive duplicate sentences from txt file)

Hello Experts and Fellow Learners! I have dozens of text files that contain monthly archives of a listserv. I'm trying to eliminate/delete quoted replies so that each message only appears once across ...
0
votes
1answer
2k views

(Win Server 2012) Can't enable Data Deduplication on C: drive

I'm trying to enable Data Deduplication on C: drive but Server Manager doesn't allow me to do so. After using PowerShell command it gave me following message: Windows PowerShell Copyright (C) 2014 ...
0
votes
1answer
262 views

I have a situation where data-deduplication in Server 2012 has corrupted the metadata - how can I copy these files from an old repository?

I have two drives - drive D: and drive F:. Both of these are drives locally attached to the Server 2012 machine. drive F is a legacy store of data migrating to a new drive (D) as it's larger. Data ...
1
vote
1answer
2k views

Batch to sort a file with removing duplicates line?

If I have a text file, I can sort it by command sort input.txt output.txt. Can I somehow also remove duplicate lines from the text file as well. (Ideally I would like a solution which can be done in a ...
2
votes
1answer
2k views

How to reduce memory usage after setting dedup=off (after having it on for a while)?

I know that setting dedup=on causes high memory usage and that it's almost never worth it (see e.g. https://serverfault.com/a/569099/208661). However I did turn it on in order to see how it works and ...
6
votes
4answers
8k views

Is there a way to extract duplicate lines in Sublime Text?

I need to perform 2 operations in Sublime Text: extract unique lines and extract duplicate lines. For example for input a b a Extract duplicates should result in: a and Extract unique should ...
2
votes
2answers
251 views

Optimizing gzipped files for block-level deduplication

I have about 100TiB of compressed gzip data on cloud volumes. On my way to run a block-level dedup (duperemove for btrfs) I find that it was compressed without -n, which causes block-level difference ...
0
votes
0answers
511 views

Enable deduplication for a single folder?

Can I enable deduplication for a single folder (and subfolders) and not for an entire volume? Note: I use Windows 10 1607 with the deduplication module installed.
0
votes
1answer
80 views

How to find extracted Zip files and their folder in mac

How can i list all zip files and their extracted folders in order to remove one of them, i downloaded lots of zip files in years(like everybody) and extracted them so now they are taking twice as ...
1
vote
0answers
319 views

How to batch delete files based on their checksum matching an existing list of checksums

I'm looking for a means of deleting files based on their checksum matching a list of previously seen checksums in a text file or some other list. The context is a large photo organization project ...
0
votes
1answer
274 views

Removing duplicates from txt file

I have a txt file that can contain up to 13.000.000 randomly generated codes seperated with "\n" by the fputcsv function in php. What is the most effective way of removing duplicates from this file? ...
0
votes
1answer
54 views

What are the odds file size remains the same if you photoshop an image?

My uncle is a photographer with lots and lots of photoshoots. He stored (duplicates of) backups across a variety of external hard drives. 90% of all backups are the same, but some have had some edits, ...
0
votes
1answer
583 views

Excel: Copy all cell values from an entire sheet to a new sheet, sort and deduplicate into a single list of unique values

The data (all text values) I have is scattered across the sheet (multiple rows and columns with gaps inbetween) and it is all calculated using formulas now I would like to make sure that I know what ...
0
votes
1answer
624 views

Find duplicate media files and auto organize, on Linux

Is there an application similar to MediaMoney (on Windows), but on Linux (Debian or Ubuntu) that given when a disk-drive or a root directory, can search across it thoroughly, identifying duplicate ...
0
votes
1answer
314 views

Remove Duplicates inconsistently choosing which records to delete

So generally when using the remove duplicates function in Excel the oldest record will be kept - and that's exactly what I want to happen! But recently when running Remove Duplicates, it is ...
0
votes
1answer
79 views

How to delete one of each set of duplicate values in Excel?

I have a spreadsheet that contains many duplicates. Instead of removing all of the duplicates, I would like to remove one of each duplicate. I want to remove one record from each set of duplicates. ...
1
vote
0answers
149 views

Using btrfs cow functionallity to rapidly and efficiently subset large files

[My aim is to use btrfs cow features to subset 30GB to 300GB files into several 10GB files without producing substantial i/o nor using additional disk space. i.e.: 300GB dna_original.fastq 10GB ...
1
vote
0answers
215 views

using btrfs features to rapidly and efficiently combine large files

My aim is to use btrfs features to combine several 3GB files into a single 9GB file without producing substantial i/o nor using additional disk space. If this works for this set of data, I plan to ...
1
vote
2answers
3k views

Removing files with same name and different extension

I have a large music collection. It is organized in the following structure: artist / album / trackTitle. Over the course of time, in some directories, I have duplicate-ish files; most of my files ...
0
votes
2answers
698 views

Undo Creating Symbolic Links by copying back the original, rename the file and deleting the link

I used a windows 7 utility to delete exact byte per byte duplicate files and replace them with a symbolic link before I knew that the symbolic link type is not allowed in our corporate domain. I need ...
20
votes
5answers
8k views

How to replace all duplicate files with hard links?

I have two folders containing various files. Some of the files from the first folder have an exact copy in the second folder. I would like to replace those with a hard link. How can I do that?
1
vote
0answers
161 views

Is in-line data de-duplication worthwhile when backing up for a photographer?

I create RAW image files plus a small selection of JPEG files derived from the RAW masters. The JPEGs once created are very, very rarely edited again but when they are because they are recompressed, ...
2
votes
1answer
120 views

How do I use an image multiple times in a OneNote document while storing the image only once?

I'm using OneNote 2013 for family history. I have created a page for each person with photos of and documents about that person. In several cases I have group photos with many family members in them....