Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [deduplication]

Deduplication is the finding and removal of duplicate files.

0
votes
2answers
24 views

De-duping a table based on an expression using one field

Mariadb: mysql Ver 15.1 Distrib 10.3.8-MariaDB I have a table built as a union of queries from several other tables. This table looks something like- +-------------+--------------+------+-----+-----...
1
vote
1answer
81 views

Combine Excel rows with duplicate keywords, but unique values in other columns

I have an Excel sheet with keywords in column B. For each row there will be a value in one cell somewhere between column E and column CR. So, line 3 will have a keyword in column B be and a value in ...
0
votes
1answer
43 views

Multiple criteria unique values, and list

I have two issues that I am looking for help with. The first is to create a dynamic list of unique ID numbers for items matching "Horse" in column J on Sheet 2!, "Ball" in column I on Sheet 2!, and ...
2
votes
0answers
692 views

In-place delta writes from a pipe

I'm looking for a way to only overwrites the parts of a file that differ from what's supposed to be written to it, so that the file is changed with the least amount of writes to disk possible. My ...
0
votes
1answer
65 views

Script for deduplicating files and folders with a particular suffix

A botched OneDrive restoration has left me with many files and folders with a " (1)" or " (2)" suffix. I would like a script (Bash is fine as I have MinGW + Cygwin, or PowerShell) that would parse ...
1
vote
2answers
187 views

How to merge duplicate folders with “name (1)”, “name (1) (1)” etc. structure

Syncing between my Google Filestream, Google Drive, and Synology CloudSync and got all messed up and I'm left with hundreds of duplicate folders with the folder name followed by a "(1)" or "(2)" etc., ...
0
votes
2answers
225 views

Is there a way to backup deduplicated NTFS drive with rsync or other linux tool?

Is there a Linux way or a tool that is aware of Windows Deduplication on NTFS? Here I use rsync to sync files from a AD file server shares. The shares are mounted, readonly, using mount.cifs on linux. ...
1
vote
0answers
26 views

Remove files in Directory B that also exist in Directory A

This page has tells how to remove files in Directory B that also exist in Directory A. I want to use the windows solution but without the copying. Can someone tell me how to edit the FOR line to ...
0
votes
1answer
113 views

rsync replace only matching files with hardlinks

How to replace files in dst/, that exactly matches by content with files from src/ with hardlinks to these (src/*) files? There is --hard-links option for rsync, which I want to use, but I don't want ...
0
votes
0answers
3k views

How to find “revision number” and appropriate Windows version to read files off an old deduped disk?

I have a disk that's come off a dead server. It's clearly deduped - the classic errors 1305, 80070519 and "bad revision" when I try to copy the files show that. I've tried every Windows Server ...
1
vote
1answer
88 views

Find sub-folders containing only duplicate files

I am looking for a method (not including paid software) for finding all folders that contain only files with that are also in at least one other sub-folder of the parent directory. If used on a music ...
1
vote
1answer
274 views

(Win Server 2012 R2) Can't enable data deduplication

Windows Server 2012 R2 (Datacentre edition) I cannot enable Data Deduplication. This particular volume previously had data deduplication enabled. Computer was changed from BIOS to UEFI boot, so ...
0
votes
1answer
1k views

Delete duplicate images with different file names

I have lots of images that have all the exact same metadata (size, the picture itself, etc) and the only difference between them is their file names. So say I have 10 images that all look the same, I ...
0
votes
1answer
85 views

How to find files that are duplicates of files in a certain folder?

I need to remove the duplicates in my computer, since i have around 5 copies of every single photo, in different folders. I want to delete the photos that also appear in a certain folder, in witch i ...
0
votes
1answer
850 views

How Borg backup handle data deduplication and ecryption?

I need a backup software which performs encryption clientside. On Borg Documentation it is stated that it performs the encryption clientside and eventually if the repository is remote, another ...
1
vote
1answer
369 views

Find duplicate hashes

I have a bunch of files (4800 files), that I believe contain many duplicates and triples. Problem is that they don't also have the same names. I already weeded out the easy ones with same names. The ...
0
votes
1answer
364 views

How do I exclude/ignore specific file types/extensions with rmlint?

I am trying to dedupe a labyrinthine directory, but I keep getting results for the contents of the items I want to match rather than the enclosing items, themselves, i.e. my results are full of things ...
1
vote
3answers
730 views

Find and delete duplicated files in different disks and directories

I have hundreds of thousands files distributed in many external disks and disks in computers and many are duplicated. This mess was caused by myself creating copies for safety proposal. From time to ...
4
votes
1answer
2k views

zfs on linux compression and dedup order

What is the order for data written to a zfs filesystem on zfs on linux? The only specific document i found at http://docs.oracle.com/cd/E36784_01/html/E36835/gkknx.html says; When a file is written, ...
0
votes
1answer
49 views

Removing Duplicates From 1 Column Assocatied with a Name in Multiple Rows [duplicate]

I'm trying to create a pivot table to count the number of events a person has attended. In the example below, I'd like to remove the duplicates associated with a specific person. Joyce is listed ...
0
votes
0answers
70 views

Removing replies in downloaded emails (i.e., non-consecutive duplicate sentences from txt file)

Hello Experts and Fellow Learners! I have dozens of text files that contain monthly archives of a listserv. I'm trying to eliminate/delete quoted replies so that each message only appears once across ...
0
votes
1answer
1k views

(Win Server 2012) Can't enable Data Deduplication on C: drive

I'm trying to enable Data Deduplication on C: drive but Server Manager doesn't allow me to do so. After using PowerShell command it gave me following message: Windows PowerShell Copyright (C) 2014 ...
0
votes
1answer
201 views

I have a situation where data-deduplication in Server 2012 has corrupted the metadata - how can I copy these files from an old repository?

I have two drives - drive D: and drive F:. Both of these are drives locally attached to the Server 2012 machine. drive F is a legacy store of data migrating to a new drive (D) as it's larger. Data ...
1
vote
1answer
1k views

Batch to sort a file with removing duplicates line?

If I have a text file, I can sort it by command sort input.txt output.txt. Can I somehow also remove duplicate lines from the text file as well. (Ideally I would like a solution which can be done in a ...
2
votes
1answer
1k views

How to reduce memory usage after setting dedup=off (after having it on for a while)?

I know that setting dedup=on causes high memory usage and that it's almost never worth it (see e.g. https://serverfault.com/a/569099/208661). However I did turn it on in order to see how it works and ...
5
votes
4answers
5k views

Is there a way to extract duplicate lines in Sublime Text?

I need to perform 2 operations in Sublime Text: extract unique lines and extract duplicate lines. For example for input a b a Extract duplicates should result in: a and Extract unique should ...
2
votes
2answers
192 views

Optimizing gzipped files for block-level deduplication

I have about 100TiB of compressed gzip data on cloud volumes. On my way to run a block-level dedup (duperemove for btrfs) I find that it was compressed without -n, which causes block-level difference ...
0
votes
0answers
406 views

Enable deduplication for a single folder?

Can I enable deduplication for a single folder (and subfolders) and not for an entire volume? Note: I use Windows 10 1607 with the deduplication module installed.
0
votes
1answer
72 views

How to find extracted Zip files and their folder in mac

How can i list all zip files and their extracted folders in order to remove one of them, i downloaded lots of zip files in years(like everybody) and extracted them so now they are taking twice as ...
1
vote
0answers
276 views

How to batch delete files based on their checksum matching an existing list of checksums

I'm looking for a means of deleting files based on their checksum matching a list of previously seen checksums in a text file or some other list. The context is a large photo organization project ...
0
votes
1answer
169 views

Removing duplicates from txt file

I have a txt file that can contain up to 13.000.000 randomly generated codes seperated with "\n" by the fputcsv function in php. What is the most effective way of removing duplicates from this file? ...
0
votes
1answer
50 views

What are the odds file size remains the same if you photoshop an image?

My uncle is a photographer with lots and lots of photoshoots. He stored (duplicates of) backups across a variety of external hard drives. 90% of all backups are the same, but some have had some edits, ...
0
votes
1answer
481 views

Excel: Copy all cell values from an entire sheet to a new sheet, sort and deduplicate into a single list of unique values

The data (all text values) I have is scattered across the sheet (multiple rows and columns with gaps inbetween) and it is all calculated using formulas now I would like to make sure that I know what ...
0
votes
1answer
552 views

Find duplicate media files and auto organize, on Linux

Is there an application similar to MediaMoney (on Windows), but on Linux (Debian or Ubuntu) that given when a disk-drive or a root directory, can search across it thoroughly, identifying duplicate ...
0
votes
1answer
148 views

Remove Duplicates inconsistently choosing which records to delete

So generally when using the remove duplicates function in Excel the oldest record will be kept - and that's exactly what I want to happen! But recently when running Remove Duplicates, it is ...
0
votes
1answer
75 views

How to delete one of each set of duplicate values in Excel?

I have a spreadsheet that contains many duplicates. Instead of removing all of the duplicates, I would like to remove one of each duplicate. I want to remove one record from each set of duplicates. ...
1
vote
0answers
129 views

Using btrfs cow functionallity to rapidly and efficiently subset large files

[My aim is to use btrfs cow features to subset 30GB to 300GB files into several 10GB files without producing substantial i/o nor using additional disk space. i.e.: 300GB dna_original.fastq 10GB ...
1
vote
0answers
191 views

using btrfs features to rapidly and efficiently combine large files

My aim is to use btrfs features to combine several 3GB files into a single 9GB file without producing substantial i/o nor using additional disk space. If this works for this set of data, I plan to ...
1
vote
2answers
3k views

Removing files with same name and different extension

I have a large music collection. It is organized in the following structure: artist / album / trackTitle. Over the course of time, in some directories, I have duplicate-ish files; most of my files ...
0
votes
2answers
585 views

Undo Creating Symbolic Links by copying back the original, rename the file and deleting the link

I used a windows 7 utility to delete exact byte per byte duplicate files and replace them with a symbolic link before I knew that the symbolic link type is not allowed in our corporate domain. I need ...
19
votes
4answers
7k views

How to replace all duplicate files with hard links?

I have two folders containing various files. Some of the files from the first folder have an exact copy in the second folder. I would like to replace those with a hard link. How can I do that?
0
votes
0answers
152 views

Is in-line data de-duplication worthwhile when backing up for a photographer?

I create RAW image files plus a small selection of JPEG files derived from the RAW masters. The JPEGs once created are very, very rarely edited again but when they are because they are recompressed, ...
2
votes
1answer
116 views

How do I use an image multiple times in a OneNote document while storing the image only once?

I'm using OneNote 2013 for family history. I have created a page for each person with photos of and documents about that person. In several cases I have group photos with many family members in them....
0
votes
1answer
93 views

Is it possible to perform Deduplication recursively?

If anybody is familiar with the concept of Deduplication (If not,read link on wikipedia) http://en.wikipedia.org/wiki/Data_deduplication Is it possible to perform Deduplication recursively i.e. ...
0
votes
2answers
89 views

In Windows, is there any software to prevent duplication of files in a folder? [closed]

Let's say I have two files c:\f1\data.txt and c:\f2\data.txt. If I issue the command copy c:\f2\data.txt c:\f1\data\ I will receive a warning about the potential overwriting. I will receive the ...
0
votes
2answers
225 views

Virtual folder combining a read-only and read-write directory (file deduplication)

Is it possible in linux to have a directory that contains all the contents of another directory (in a read-only fashion) but you are able to write to the directory without it affecting the 'parent' ...
17
votes
3answers
1k views

How to deduplicate 40TB of data?

I've inherited a research cluster with ~40TB of data across three filesystems. The data stretches back almost 15 years, and there are most likely a good amount of duplicates as researchers copy each ...
1
vote
1answer
1k views

Deleting all but latest revision of a file

So I have these massive lists of drawings that I've done for work and I'd like to be able to dump them all into one folder and run a batch that would delete all of the older revs and leave the highest ...
0
votes
1answer
692 views

ZFS + dedup: copy many small files fast

I'm running a ZFS backup server (with deduplication) to backup websites and store them with retention. To do this, I transfer all files to the backup server during the first backup, and make a copy of ...
11
votes
4answers
91k views

How to remove duplicate rows based on some columns

I have a excel sheet which contains duplicate rows I want to remove a row if its A C D E F columns are same (Ignore B while calculating duplicates but remove it while removing a row) At the moment ...