How do I use ImageMagick convert in a reproducible manner (without timestamps)?

Question

I am trying to use ImageMagick in a script to convert and resize a large number of files for version control. I need the files to convert the same time every time so git won't commit files that just had updated timestamps. Unfortunately, ImageMagick insists on adding a create and modify timestamp to each image that makes git recommit every file again.

I have searched a lot about this issue and I have tried the following flags:

-define png:exclude-chunks=date
+set date:create +set date:modify
-strip

None of these have resulted in a reproducible process:

-define png:exclude-chunks=date

stephen@Saturn ~/test (git)-[master] % convert input.png -define png:exclude-chunks=date -resize 100x100 1.png
stephen@Saturn ~/test (git)-[master] % convert input.png -define png:exclude-chunks=date -resize 100x100 2.png
stephen@Saturn ~/test (git)-[master] % diff 1.png 2.png
Binary files 1.png and 2.png differ
stephen@Saturn ~/test (git)-[master] % cmp -l 1.png 2.png
  125  41  42
  126  67   0
  127 322 101
  128 321 101
  129  35 353
  130  64 370

+set date:create +set date:modify

stephen@Saturn ~/test (git)-[master] % convert input.png +set date:create +set date:modify -resize 100x100 1.png
stephen@Saturn ~/test (git)-[master] % convert input.png +set date:create +set date:modify -resize 100x100 2.png
stephen@Saturn ~/test (git)-[master] % diff 1.png 2.png
Binary files 1.png and 2.png differ
stephen@Saturn ~/test (git)-[master] % cmp -l 1.png 2.png
  125  51  52
  126  71   0
  127 375 211
  128 260 230
  129 272 141
  130  73 360

-strip

stephen@Saturn ~/test (git)-[master] % convert input.png -strip -resize 100x100 1.png
stephen@Saturn ~/test (git)-[master] % convert input.png -strip -resize 100x100 2.png
stephen@Saturn ~/test (git)-[master] % diff 1.png 2.png
Binary files 1.png and 2.png differ
stephen@Saturn ~/test (git)-[master] % cmp -l 1.png 2.png
  110  41  45
  111 241 246
  112 235 360
  113 264 160
  114 252 263

How can I accomplish reproducible converts with ImageMagick?

Answer 1

You need to update ImageMagick to version 6.9.1-3 or above, and then all of the commands from your question will create reproducible images.

I found the following in the changelog:

2015-04-20  6.9.1-3 Cristy  <quetzlzacatenango@image...>
  * Support -define compose:clamp=false option (reference
    https://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=26946).
  * Don't extend any user supplied image buffer in SeekBlob() (bug report
    from a.chernij@corp...).
  * Improved reproducible builds (reference
    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=783933).
  * Draw a rectangle of width & height of 1 (reference
    https://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=24874).

I found the changelog entry after discovering that ImageMagick on ArchLinux works fine, contrary to ImageMagick on Ubuntu 16.04.

ArchLinux (good, reproducible images):

$ convert --version
Version: ImageMagick 6.9.8-8 Q16 x86_64 2017-05-30 http://www.imagemagick.org

Ubuntu 16.04 (bad, different images every time):

$ convert --version
Version: ImageMagick 6.8.9-9 Q16 x86_64 2017-05-26 http://www.imagemagick.org

Answer 2

I have given up on getting ImageMagick to behave, and I have resorted to using GraphicsMagick, which appears to have all the features of ImageMagick, minus this timestamp bug:

stephen@Saturn ~/test (git)-[master] % gm convert -resize 100x100 input.png 1.png
stephen@Saturn ~/test (git)-[master] % gm convert -resize 100x100 input.png 2.png
stephen@Saturn ~/test (git)-[master] % diff 1.png 2.png
stephen@Saturn ~/test (git)-[master] % cmp -l 1.png 2.png

Identify shows 2 differing timestamps, but it's getting it from the file properties instead of embedded metadata, and diff / cmp shows the files as being identical.

Answer 3

I found -define png:exclude-chunks=date,time was necessary. Excluding just the date chunk was not enough.

That single options was also sufficient; I did not need -strip or exporting a specific timestamp using SOURCE_DATE_EPOCH (which has been suggested elsewhere).

Finally for viewing the difference diff <(xxd 1.png) <(xxd 2.png) was my preferred way, to see both binary and ascii (helpful to see chunk ID and where a timestamp appearing).

For reference, just excluding date:

8,9c8,9
< 00000070: 0000 0774 494d 4507 e503 030a 1d0f bd0c  ...tIME.........
< 00000080: 01f4 0000 8000 4944 4154 78da ecdd 7578  ......IDATx...ux
---
> 00000070: 0000 0774 494d 4507 e503 030a 1c19 50c3  ...tIME.......P.
> 00000080: 85e4 0000 8000 4944 4154 78da ecdd 7578  ......IDATx...ux