When putting two identical files in a .zip, they take 2x space, can this be avoided

Question

Say that you have a file you want to put in a .zip archive:

zip a1.zip foo.dll

My test .dll file is ~10MB, and the archive turns out to be 3.5MB

Then you create a file with the exact same contents, and put these both into an archive:

cp foo.dll bar.dll
zip a2.zip foo.dll bar.dll

You might expect that ZIP is smart enough to figure out this is repeating data and use only one compression object inside the .zip, but this is not the case: a2.zip is 7.0MB!

Basically most such utilities behave similarly (tar.gz, tar.bz2, rar in solid mode) - only 7zip caught me and the resulting a2.7z is only marginally larger than a1.7z.

So the question is: is it possible to construct a .zip file where this space wastage is avoided? We create the .zip files with C++ code, which uses the minizip project from zlib.

---

Why do we need this?

We ship our software in both ".exe installer" and ".zip file" form. The software doesn't really require installation, you can just unzip and use it. The .zip option is preferred by large clients which have many workstations and use automated deployment / software update services.

We introduced three .dll files recently, that now need to be put in two different folders, to be used by different components (only one central directory for these files is impossible for technical reasons). This three .dll files are exact copies in both folders. The .exe installer figures this out, since we instruct it to use the exact same compressed blob for each of the two destinations. But that's not the case with .zip and the resulting install is 15MB larger, meaning more bandwidth usage, slower download times and searing engineer-unhappy-that-things-are-not-optimal type of rage. Also, the .zip install suddenly becomes larger than the .exe install, so we'd be asked what did we omit in the .exe install.

There are some potential solutions to this,

• Use 7-zip: the boss is strongly against this, however, since this forces the aforementioned automated deployment people to modify their scripts to accommodate 7-zip.
• Use symlinks: if you put a symlink inside a .zip, that points to another file inside the .zip, it is stored as a reference (e.g., by using the --symlinks option to zip). Hopefully, unarchivers under Win32 will be kind enough to support these and extract a copy of the file to the path where the symlink should be located. F.e. WinRAR does this, but there are a lot of programs that "can do .zip" and I'm not sure if all do it this way.

Answer 1

You can put the two files into an uncompressed zip file (e.g. with 7-Zip), then put the resulting file into a zip file again.

Answer 2

You can solve your issue by writing a small un-archiving module. You can either distribute this un-archiver to users, so they use it to extract the zip, or better still, implement the module as part of your .EXE program itself. The module could be C# console program with something like this:

        private static void Extract(string filename)
        {
            //ZipInputStream zi = new ZipInputStream (File.Open ("", FileMode.Open));
            using (ZipInputStream s = new ZipInputStream(File.OpenRead(filename))) {

                ZipEntry theEntry=null;
                while ((theEntry = s.GetNextEntry()) != null) 
                {

                    Console.WriteLine(theEntry.Name);

                    string directoryName = Path.GetDirectoryName(basedir + "ext" + Path.DirectorySeparatorChar + theEntry.Name);
                    string fileName = Path.GetFileName(basedir + "ext" + Path.DirectorySeparatorChar  + theEntry.Name);
                    Console.WriteLine("And the path is:" + basedir +  "ext" + Path.DirectorySeparatorChar  + theEntry.Name);

// create directory
                    if ( directoryName.Length > 0 ) {
                        //Console.WriteLine("DIRECTORY IS SOMETHING");
                        Directory.CreateDirectory(directoryName);
                    }

                    if (fileName != String.Empty) {
                        using (FileStream streamWriter = File.Create(basedir+"DLL_PATH" + Path.DirectorySeparatorChar  + theEntry.Name)) {

                            int size = 2048;
                            byte[] data = new byte[2048];
                            while (true) {
                                size = s.Read(data, 0, data.Length);
                                if (size > 0) {
                                    streamWriter.Write(data, 0, size);
                                } else {
                                    break;
                                }
                            }
                        }
                    }
                }
                  System.IO.File.Copy("source.dll", "destination.dll"); //IMPORTANT

            }

        }

This is not tested, but I'm sure you can work with it. The most important part is this where you copy the extracted file to the new dll:

System.IO.File.Copy("source.dll", "destination.dll"); //IMPORTANT

Make sure you include the opensource SharpZipLib DLL with this and these are the namespaces used:

using System;
using ICSharpCode.SharpZipLib;
using ICSharpCode.SharpZipLib.Zip;
using System.IO;
using System.Collections;
using System.Collections.Generic;
using System.Xml;

Answer 3

I propose two alternatives:

• distribute a self-extracting .exe (not an installer), which creates a directory in which all files are in the right relative place (then the client just have to drag and drop the directory where he wants). If you use 7zip to create that you would have the advantage of smaller space, moreover it can be opened as an archive by 7zip itself if people have that installed.
• embrace simplicity and stay with the plain .zip with duplicated files. Is bandwidth that much of a problem? Wouldn't users complaining about stuff not working as expected/not being able to perform a complicated install procedure be a bigger hassle? From the client point of view it is probably a no-issue: 16 MB is usually not more than 1 minute download time, and it is not like they have to download the same file every day. Of course it depends on the size of the whole installer: if the difference is between 5 MB and 21 MB I would worry about it too.