Deduplication is the finding and removal of duplicate files.
0
votes
0answers
13 views
Background way to find duplicate media files [duplicate]
Any know a way to semi-automatically (in background?) identify (exact) duplicate media files to simplify deduplication? Media files eat up server space yet the need ot avoid loss leads to multiple ...
1
vote
1answer
57 views
What is the best way to merge two sets of files, but for duplicates to select the older file in Windows?
I have stumbled into a dumb usage case. I want to merge two folders in a particular way (I guess these are called folders now, I still think of them as directories, but whatever). The folders have ...
1
vote
1answer
90 views
Deduplicate Files inside multiple .tar.gz archives
I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB).
I would like to archive these files and deduplicate (solid compression) files ...
0
votes
0answers
72 views
Copy only files that don't exist regardless of file name?
My dropbox account apparently renames all auto uploaded images. I'm about to clear my phone and I would like to verify that all images are in the dropbox folder before deleting them.
If they had their ...
0
votes
1answer
53 views
Checking files on an external drive to whole computer for duplicates
I'm looking to use an external drive for a new purpose and want to see which, if any, files on it are not already on my main pc.
I've seen comparison programs that can compare folders or one drive to ...
1
vote
1answer
478 views
View ZFS deduplication ratio on a dataset
I have a tank consisting of several datasets, only one of which is configured to use deduplication.
How can I see the ratio for this dataset? I get a ratio of 1.00x for the whole pool but I imagine ...
0
votes
0answers
20 views
Storing zsync-like differences for MSDN downloads?
Has anyone been able to create something like the zsync files that are available for Ubuntu CD/DVD images for those images downloaded from MSDN?
We are storing numerous copies of the various editions ...
1
vote
4answers
110 views
Duplicate name exists solution
I have about 70 pc's with exactly same hardware. I decided to automate turning on and off.
I took 1 PC. Here is what I've done:
Changed bios configuration so that now pc's waking when I turn on AC ...
0
votes
0answers
33 views
Outlook 2003 Duplicating emails, even while “working offline”
I've a user running Outlook 2003 on an XP machine. Yesterday, the inbox exploded.
Thousands upon thousands of messages in the inbox. By the time I was notified, there were nearly 60,000 messages. ...
2
votes
1answer
185 views
ZFS compressed and deduped filesystem on file
I can create squashed filesystems with mksquashfs in this way:
mksquashfs dir-1 dir-2 ... dir-n filesystem-on-file.squash -comp xz -Xbcj x86 -Xdict-size 1048576 -b 1024k
mksquashfs provides ...
2
votes
3answers
190 views
Fuzzy Duplicate File Finder?
Is there any GUI application (either for Windows or Linux) that implements fuzzy hashing algorithms like ssdeep or sdhash to find near duplicate files (mostly text files)?
0
votes
0answers
70 views
De-duplicate folders of PDFs based on image similarity? [closed]
On my Windows server, I have multiple folders containing hundreds if not thousands of PDFs (primarily documentation scans, scanned as images, non-OCR'ed).
I'm consolidating and reorganizing and need ...
4
votes
1answer
266 views
How to find duplicate images and compare their meta-data?
Is there an application for Windows that finds duplicate images and lets quickly preview the image and, what is most important, the meta-data of the image files.
I can see there are programs for ...
3
votes
1answer
77 views
Painless way of consolidating files across multiple machines/OSes
Just bought a NAS. So I thought I'd get all our photos, media files and pdfs consolidated, de-duplicated, de-junked and virus-checked and stick them all on it.
We have 3 laptops, one running Windows, ...
1
vote
0answers
208 views
How to deduplicate files when storing in the SDFS file system using OpenDedup
I followed the link, http://www.opendedup.org/quickstart for
installing OpenDedup in my linux machine (Ubuntu 12.04 - 64 bit). I created a SDFS volume and
mounted it.
I copied two similar Ubuntu VMs, ...
1
vote
3answers
272 views
Program for remove exact duplicate files while caching search results
We need a Windows 7 program to remove/check the duplicates but our situation is somewhat different than the standard one for which there are enough programs.
We have a fairly large static archive ...
2
votes
1answer
209 views
How to de-dupe identical photos that have a slightly different file size?
I imported many photos using the new "camera import" feature of Dropbox. Many of those were duplicates of photos previously imported by direct copying from the camera.
Strangely, the Dropbox import ...
1
vote
1answer
183 views
Is there a way to remove duplicate emails from a remote account?
I have a user who has multiple duplicated emails across multiple folders on his IMAP account. How he managed to create them is beyond me, but mine is not to reason why, just to fix it!
Can anyone ...
1
vote
1answer
128 views
File synchronising software
Throughout my PC I have to have exactly the same files in more than one location (for one reason or another) and it's getting really difficult to keep track of them all and keep them all up-to-date ...
1
vote
1answer
416 views
Detecting image duplicates in 7-Zip archives on Windows?
I have a large image gallery, and a few years later I realized that I get more and more duplicates. :) Basically, my gallery consists of 7-Zip archives containing the image sets. I saw there are some ...
0
votes
1answer
132 views
Find RARs with duplicate content
I need a utility to find RAR files that contain duplicate data (i.e. files within the RAR that hash the same, but could have different names).
I can open the RARs and see the CRCs are the same, but I ...
2
votes
2answers
314 views
Opensource duplicate finder for directories?
Is there an open source tool that will tell me if two directories have byte-for-byte identical content? I need a tool that confirms if two directories are the same. A listing of files that I need to ...
1
vote
2answers
893 views
Multicolumn spreadsheet: extract unique values in Column 1, given same values in Columns 3 and 4?
Here's a problem: given a (large) spreadsheet with userIDs in Column 1, a time/date pair in Column 2, a course code in Column 3, and a course title in Column 4: I want to produce a report showing how ...
1
vote
1answer
439 views
How can I consolidate duplicate entries in iTunes without losing statistical data?
I recently combined two of my PCs in a Frankensteinian attempt to optimize my hardware configuration. One of these PCs was much older and had a different collection of music on it than my current one ...
7
votes
3answers
1k views
Is there a compression or archiver program for Windows that also does deduplication?
I'm looking for an archiver program that can perform deduplication (dedupe) on the files being archived. Upon unpacking the archive, the software would put back any files it removed during the ...
1
vote
1answer
433 views
Why is de-dupe feature of ZFS not working with FreeBSD 8.2
I have FreeBSD 8.2 installed on AMD/64 with 3 GB RAM and 690 GB pooled HD under ZFS. I have two copies of an identical file (180 MB video) in same directory and yet df -h indicates that both are using ...
1
vote
1answer
508 views
Windows 7 file-based backup service
I'm looking for a good replacement for Lazy Mirror, since it doesn't support Windows 7 well.
Pros:
One of the things I really loved about Lazy Mirror is that it always maintains a "full" backup, but ...
1
vote
1answer
389 views
Looking for a HDD tool that can perform deduplication and data compression
Hi experts and pros! And a happy new year 2011 to everyone!
I'm looking for a "private" tool that could deduplicate my external HDD. I'm sure I have tons of files double on my Disk drives, and I ...
1
vote
3answers
140 views
Find copies of folders? (Not files)
I have a dozen of folders that are duplicates. Within them are a few dozen folders that are duplicates so i have a few thousand copies of the same files and folders. Many of them are exactly the same ...
5
votes
4answers
203 views
Locate near-duplicate files in a folder
I have a folder full of files, with a number of duplicate files in it. Unfortunately, in a number of cases, one version is an updated version of another, so a straight byte-match doesn't locate the ...
1
vote
4answers
382 views
Tool for merging multiple OFX files into one?
OFX is an XML format for financial transaction history and many banks, including mine, provide the option to download your transaction data in this format. Unfortunately, they often force you to ...
2
votes
1answer
561 views
Remove duplicate emails (Outlook 2003)
Sometimes our POP server has a fit and it re-sends a bunch of e-mail messages again to a particular user and I have to clean up those duplicate messages that ended up in the local outlook store.
Is ...
1
vote
3answers
319 views
Backup of 5TB server
I'm looking for a software that will help me backup a server with 5TB of data. My wishlist:
Backup on 1-2TB SATA disks
Should need little to no configuration (just the directories to backup and the ...
0
votes
0answers
191 views
How can I remove duplicate messages from my inbox? [duplicate]
Possible Duplicate:
Removing duplicate emails in Thunderbird
For some reason known only to Thunderbird and Hotmail they decided to re-download all my e-mails, twice. So I now have about 20k ...
6
votes
5answers
825 views
Ways to deduplicate files
I want to simply backup and archive the files on several machines. Unfortunately, the files have some large files that are the same file but stored differently on different machines. For instance, ...
0
votes
3answers
375 views
Deduplication of two sets of contacts
One of our users has two sets of contacts in his email: 7000 or so from and old mail system, and even more in his current Apple Address Book. What we're trying to do is get the two sets of contacts ...
9
votes
5answers
5k views
Free Duplicate mp3 finder
Sometime back I had used a duplicate file finder for mp3 by analyzing the content.Unfortunately it was not free and the shareware had a lot of limitations.
Are there any freeware/OSS ones to detect ...
42
votes
15answers
8k views
Which duplicate files and folders finders exist for Windows? [closed]
I need a free duplicate file finder/remover app, with ability to find duplicate files/folders by name and/or by size and to remove one of duplicates.
