Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a directory with subdirectories, and a lot of duplicate files in them. If I move everything to a single rar archive, will WinRAR detect the duplicate files, or will all of them be archived and add up to the size of the rar archive?

share|improve this question
I do not think WinRar does such intelligence because not only the filenames and extensions could be same in different dir heirarchy, their contents will also be different/same. So checking each and every byte in it is costly and difficult. One test you can do is, carry out the first as said above and see size. Change 1 file name and do same and see size again. It shud be same. – Zenwalker Mar 29 '12 at 10:16
As with any archive program it will archive exactly what you tell it to do. If you want to get rid of duplicates do that before you create the archive. – Ramhound Mar 29 '12 at 12:28
up vote 3 down vote accepted

The new version of WinRAR 5.x has introduced new RAR5 archive format and this is one of many improvements:

Save identical files as references If this option is enabled, WinRAR analyzes the file contents before starting archiving. If several identical files larger than 64 KB are found, the first file in the set is saved as usual file and all following files are saved as references to this first file. It allows to reduce the archive size, but applies some restrictions to resulting archive. You must not delete or rename the first identical file in archive after the archive was created, because it will make extraction of following files using it as a reference impossible. If you modify the first file, following files will also have the modified contents after extracting. Extraction command must involve the first file to create following files successfully.

It is recommended to use this option only if you compress a lot of identical files, will not modify an archive later and will extract an archive entirely, without necessity to unpack or skip individual files. If all identical files are small enough to fit into compression dictionary, solid archiving can provide more flexible solution than this option.

Supported for RAR 5.0 archives only.

My quick test on folder that contains 320000 files (Baldur's Gate Trilogy + Lot of mods) look like this:

RAR compression method, compression set to "Store": 26.1 GB (28,053,815,768 bytes) RAR5 compression method, compression set to "Store": 23.9 GB (25,722,664,097 bytes)

So I was able to save over 9% without using any compression at all!

share|improve this answer

If the files are really duplicates (or near duplicates), compression software can exploit that similarity across files to greatly increase the compression ratio. It's called Solid Compression. WinRAR and 7-Zip are 2 popular archivers that use it -- 7-Zip does by default. I'm not a RAR user so I can't tell you it's default configuration.

Common archivers on Linux/Unix/BSD systems also implicitly do solid compression by concatenating all the files together into a single file (most often via tar) before compressing that single file as a large block.

The one giant caveat to all this is that you don't really have any way of knowing exactly which files are similar, or how similar they are. It's not a good way of finding out what duplicate files you have, and extracting the archive is going to restore all that duplication. Which is, normally, exactly what one wants and expects with data compression -- to get back out exactly what was put into it.

If you want to clean up your folders, you need duplicate detection software. For normal collections, there's tons of software out there that ferrets out duplicate files. If you're dealing with media (audio, video, pictures), then you're going to want software that doesn't search for exact duplicates, but can fingerprint your files and find groups of files that are similar. That way, if you've got 2 copies of the same song with different tags or compressed slightly differently (say, a 128 Kb/s MP3 and a 256 Kb/s AAC) they can be identified. Or identifying 2 pictures of the same subject where one has been cropped or edited. Each media type often has specialized software for finding similar files, and there have been questions here before dealing with the particulars of each type. Of course, cleaning up such collections is much more difficult and time consuming because there's no fast and easy rules for deciding which file should be kept.

share|improve this answer
Thanks for the clear explanation on solid compression! I just tried 7-zip on a directory with a lot of redundancy, and got a factor 10 improvement compared to zip -- although only after significantly increasing dictionary size and word size from their default values (default parameter values yield basically no improvement). – mitchus Apr 4 '13 at 13:10

WinRAR will not do what you want. However, there are other tools that can find duplicated files inside a folder or in a partition. I have needed to do such a thing before, and I used Easy Duplicate Finder software:

Easy Duplicate Finder is a powerful tool to find and resolve duplicate photos, documents, spreadsheets, MP3's, and more! Removing duplicates will also help to speed up indexing and reduces back up size and time. Your computer isn’t fully optimized until you’ve removed all unnecessary duplicate files. Let Easy Duplicate Finder remove the duplicates!

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .