I just did a little experiment where I created a tar archive with duplicate files to see if it would be compressed, to my awe, it was not! Details follow (results indented for reading pleasure):
$ dd if=/dev/urandom bs=1M count=1 of=a 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.114354 s, 9.2 MB/s $ cp a b $ ln a c $ ll total 3072 -rw-r--r-- 2 guido guido 1048576 Sep 24 15:51 a -rw-r--r-- 1 guido guido 1048576 Sep 24 15:51 b -rw-r--r-- 2 guido guido 1048576 Sep 24 15:51 c $ tar -c * -f test.tar $ ls -l test.tar -rw-r--r-- 1 guido guido 2109440 Sep 24 15:51 test.tar $ gzip test.tar $ ls -l test.tar.gz -rw-r--r-- 1 guido guido 2097921 Sep 24 15:51 test.tar.gz $
First I created a 1MiB file of random data (a). Then I copied it to a file b and also harlinked it to c. When creating the tarball, tar was apparently aware of the hardlink, since the tarball was only ~2MiB and not ~3Mib.
Now I expected gzip to reduce the size of the tarball to ~1MiB since a and b are duplicates, and there should be 1MiB of continuous data repeated inside the tarball, yet this didn't occur.
Why is this? And how could I compress the tarball efficiently in these cases?