Tag Info

Hot answers tagged

18

I have tried literally dozens of duplicate file finders (I still have the installers/ZIP files for about 20 of them sitting around). I used CloneMaster 2.19 for a time because it was the best one I could find, though even that wans’t perfect (I wanted one that could also detect duplicate MP3s by audio content, ignoring the tags). All of them had problems ...


14

Duplicate Cleaner is very fast and it has extensive result set filtering possibilities.


9

There was this Duplicate File Finder available for some time but now its only on secondary sites, like this Softpedia reference. The Wareseeker site shows the correct reference http://dff.nazrashid.com/ which is no longer around. I would be careful fetching executables from such sites though. It can do search for duplicates within multiple directory trees ...


8

Almost all modern archivers do exactly this, the only difference is that they refer to this as a "solid" archive, as in all of the files are concatenated into a single stream before being fed to the compression algorithm. This is different from standard zip compression which compresses each file one by one and adds each compressed file to the archive. ...


7

Clonespy http://www.clonespy.com I find this very helpful and more useful than doublekiller. A particularly nice feature is the "Pools" feature, where you can compare one group of directories with another group of directories while ignoring duplicates within the groups. For example, my partner likes to keep duplicate image files while working with images. ...


6

Try MediaMonkey. If you read this thread on Lifehacker it comes highly recommended. It has scripts for identifying duplicates by metadata and hash values. The free version of MediaMonkey does all you need. The advantage of using MediaMonkey is that you can do a whole lot more with it. You can add metadata info to your music, find and add album art (it looks ...


6

I have the following batch file lying around for some time: @ECHO OFF REM TODO: Help when run with /? and switch for recursion REM Furthermore check whether we might have enough files to hit the envvar length limit REM and switch strategies accordingly (slower but finds all dupes then) SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION REM This method speeds ...


6

It's probably an issue with localization. There's a desktop.ini file in the C:\Users\Public\Pictures\Sample Pictures folder, that has some lines like these [LocalizedFileNames] Chrysanthemum.jpg=@%systemroot%\system32\SampleRes.dll,-101 The 8 files you listed in your question are the ones that are listed in my copy of this desktop.ini file. So my guess ...


6

The disk is an NFS mounted disk. When I go to the host computer that publishes the drive, the file is only listed once. Probably a bug, issue, or race condition with NFS. It's possible to have two files of the same name if you directly edit the filesystem structures using a hex editor. However I'm not sure what would happen if you try to delete or ...


6

A friend helped me track this down and found this is a bug as recorded in Bugzilla 38572 for the Linux kernel here. The bug is supposedly fixed in version 3.0.0 of the kernel, but present at least in version 2.6.38. The issue is that the server's ReadDIR() RPC call returns incorrect results. This occurs because of the following: When the client reads a ...


5

What you're referring to as a tank is really a ZFS pool and your datasets are ZFS filesystems within the pool. ZFS deduplication has pool-wide scope and you can't see the dedup ratio for individual filesystems. If you turn dedup on for a pool that already contains data, the existing data will not be automatically deduped and your ratio will still be 1.00x. ...


5

From what I recall in Windows 7, the Libraries are something particular, which regroups the content from several folders. They do not behave like your regular folder. In your particular case, it is highly possible that you have in there images from "all users", and your main account, with the same names. It doesn't explain why you would have different ...


5

Not with Windows as it normally works. You'd need something else to pull that particular stunt off. If these files are being stored on a server, Windows Storage Server has a Single Instance Store feature that can give you something close to what you want. OpenDedup is a cross platform file system implementation that does dedupe as well. You could run it on ...


4

The following works on Linux and on Cygwin. vimdiff <(cut -c8-19,23- file1) <(cut -c8-19,23- file2) For some reason on Cygwin, Vim prompts that each file has changed since editing started and asks, "[O]K, (L)oad File:". Just type O each time. Granted you don't get to see the omitted columns, but it is a visual, side-by-side comparison.


4

This will create a 100 MB file on /tmp then a zpool and zfs file system on it with both compression and deduplication enabled: cd /tmp dd if=/dev/zero of=zfile bs=1024k count=100 zpool create -O dedup=on -O compress=on filepool /tmp/zfile The file system is mounted on /filepool and unlike squashfs is writable. Just copy your files and directories in it ...


4

There is a chance that you have a hidden non-printable character or whitespace in one of the filenames. You can check with by providing the -b option to ls, e.g.: user@server:~/test$ ls -lab total 8 drwxr-xr-x 2 user user 4096 Sep 3 12:20 . drwx------ 8 user user 4096 Sep 3 12:20 .. -rw-r--r-- 1 user user 0 Sep 3 12:19 hello -rw-r--r-- 1 user user ...


4

Total Commander has a brilliant duplicate file finder utility. Unfortunately it is not free. As nik mentioned there is the Duplicate File Finder which is open source and cross platform. It's available on SourceForge here


4

Assuming all lines are shorter than 7kB, and that you have bash, dd, tail, head, sed and sort installed from cygwin/unix: { i=0 while LANG= dd 2>/dev/null bs=1024 skip=${i}000 if=large_text_file count=1021 \ | LANG= sed -e '1d' -e '$d' | LANG= sort -u ; do i=$((1+$i)) done LANG= dd 2>/dev/null bs=1024 skip=${i}000 if=large_text_file ...


4

Or, is there a way to direct a process to use disk-mapped RAM so there's much more available and it doesn't use system RAM? Yes, It's called the swap drive. You probably already have one. If you're worried about running out of RAM then increasing this is a good place to start. It works automatically though so there is no need to do anything special. I ...


3

I love this program. It's a free utility called Fast Duplicate File Finder. It finds duplicate files in a folder and all its sub folders by analysing the data. The internal preview supports images, videos, music, text and binary files. That will help you find duplicate duplicate songs. That may be all you need. But if your duplicate music is in a ...


3

Winmerge http://winmerge.org/ Compare very similar directories, perhaps between your "Documents" folder and your impromptu "backup", "Copy of Documents" folder, you created several weeks ago "just in case". You can figure out what files are still identical, which files are the newest (or oldest, if you have encountered a problem), and which files exist ...


3

EasyTAG can do this all and more easily. Most importantly for your purposes, it allows you to define and use a naming scheme based on the ID3 tags. Edit: After rereading the question, this isn't exactly what you wanted. However, after renaming the files, you can use FSlint to delete the duplicates.


3

DoubleKiller is an excellent free (they also have a pro version) duplicate file finder/remover. I've been using it for years and would thoroughly recommend it. http://www.bigbangenterprises.de/en/doublekiller


3

There is no point in using deduplication with a compression process. Most compression algorithms create what is called a 'dictionary' that will look for most common, or reused bits of data. from there it will just reference the dictionary entry instead of writing the whole "word" over again. In this way most compression processes already cut out redundant or ...


3

Gmail has an automatic duplicates detector. You could: Set up a new account Import both sets of contacts Use the Gmail duplicates finder to delete and merge them Export into a master contact list. Your user would also get a bonus Gmail account out of it...


3

Create an md5sum of each file, duplicates md5sums suggest (but doesn't guarantee) duplicate files.


3

From looking at the available Linux clients for WebDAV, my own preferred method of doing this would be: Use GVFS or one of the WebDAV filesystem modules (davfs2 or fusedav) to "map" the remote WebDAV server's files into the local filesystem path. Use the built-in cp command to with the -n option to instruct it to "not clobber" files in the destination. ...


3

There are many tools which accomplish what you want. You can use the Advance System Care. I use it and its much pretty cool tool. It has many feature and of course what are you looking for. You can also use Duplicate File Finder. All features are listed there.


3

I've recently had to do similar pruning. DupeGuru (the PE edition) worked very very well at finding dupes and similar files. I was able to delete several gigs of dupes that would have been impossible to determine and work through manually. Good luck!


2

Aproach Choose a collision-free hash function. My example uses SHA1, since the bottleneck is going to be the hard drive anyway. If that takes too long, it would be possible to compare only the first megabyte of the files. That should be enough for images. Read the files of interest on the disks A..M, compute their hashes and store them in a file specific ...



Only top voted, non community-wiki answers of a minimum length are eligible