docker load: invalid diffID for layer

Question

I am trying to transfer a docker image between two machines:

• machine 1
  • ubuntu 16.04
  • docker 17.06.2
• machine 2
  • ubuntu 18.10
  • docker 18.03.1

This is what I did:

-machine 1:

docker save -o /path/to/usb/image.tar image

-USB stick transfer from machine 1 to machine 2

-machine 2:

$ docker load -i /path/to/usb/image.tar
36018b5e9787: Loading layer [==================================================>]  200.1MB/200.1MB
invalid diffID for layer 0: expected "sha256:36018b5e978717a047892794aebab513ba6856dbe1bdfeb478ca1219df2c7e9c", got "sha256:a81b174512918f17d0735e6c32075c2437c22fac6b13c1d20c92449406f66bcd"

Does someone have an explanation for this?

Investigations

It could be a problem with my USB stick, but:

• I tried this twice, so I doubt it is due to a wrong copy
• I performed a diff between image.tar on machine 1 & image.tar on USB stick, no difference
• I use this USB stick intensively nearly everyday, it works fine

---

Also, after having copied the image archive on the USB stick, I am able to load it back onto machine 1.

---

Finally, I tried to perform the docker load by reading from stdin instead from file, still with no success:

$ cat /path/to/usb/image.tar | docker load
invalid diffID for layer 0: expected "sha256:36018b5e978717a047892794aebab513ba6856dbe1bdfeb478ca1219df2c7e9c", got "sha256:a81b174512918f17d0735e6c32075c2437c22fac6b13c1d20c92449406f66bcd"

---

It works fine with machine 3, ubuntu 16.04, docker 18.05.0.

---

I upgraded to docker 18.06.0 on machine 1, still no luck.

Answer 1

The diffID error comes from comparing the manifest shipped in the tar file with the digest of the actual bytes of the layer tar file that docker is trying to import.

A few things to look at:

Check for something modifying the exported tar file itself, e.g. something treating the file as ascii text and modifying line endings. You can try compressing the file with zip or gzip before transferring it which may prevent or at least reveal corruption in transit.

Low disk space resulting in partial writes. If it's running low on disk space in the /var/lib/docker directory, you'll get errors from docker. This could happen (many installs default this to the same as the / disk and do not allocate much disk space).

Corruption in docker's overlay2 filesystem on the source. If this happens, my instinct is to start fresh: stop docker (systemctl stop docker), completely remove /var/lib/docker (rm -rf /var/lib/docker which will delete all containers, images, volumes, so backup what you need), and then restart docker (systemctl start docker). You'll need to download the base images again and rerun your build (hopefully from a Dockerfile).

If possible, I'd consider using a registry server instead of USB to transfer images. You can run your own from the registry:2 docker image.

Answer 2

Sounds like there could be a missmatch in the diffID's on your system. The layers have a UID associated with them which is stored in several locations, two of which are:

var/lib/docker/image/overlay2/distribution/v2metadata-by-diffid var/lib/docker/image/overlay2/distribution/diffid-by-digest

these store the SHA256 for the diffid. I am not positive that these locations are used in the check which is giving you the error, but might be worth investigating for the diffID or hash throwing the error

Answer 3

If you extract the image tar, you see each layer in a dir, where a layer.tar is present. That is the content of each layer. The diffID is coming from sha256sum layer.tar. I guess something was modified during the process of copying. Try save and load again, maybe disk problem or so.