Docker/OSX can't find reason for "no space left on device"

Question

Env versions:

macos 10.14.4
docker server 18.09.2
docker desktop Version 2.0.0.3 (31259)

Faced with no space left on device error inside container and can't determine where does overflow come from.

The worst thing is I can't determine actual container size from the mother host system point of view (info/inpsect/system df tools), in fact stats from docker and mother hosts does not equal :/

I have a container with following df stats inside it:

   root@31a71014ad95:/usr/local/app# df -h
   Filesystem      Size  Used Avail Use% Mounted on
   overlay          63G   21G   40G  34% /
   tmpfs            64M     0   64M   0% /dev
   tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
   osxfs           234G  182G   48G  80% /shared_from_host
   /dev/sda1        63G   21G   40G  34% /etc/hosts
   shm              64M     0   64M   0% /dev/shm
   tmpfs           2.0G     0  2.0G   0% /proc/acpi
   tmpfs           2.0G     0  2.0G   0% /sys/firmware

(shared_from_host - is shared volume for intepoeration with mother host)

Should be mentioned I'v read about general image file size on macos, it seems ok and far from default 64Gb size:

ls -lah ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
-rw-r--r--@ 1 coin  staff    21G May 26 22:25 /Users/usss/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2

Docker container definetaly contains a big subdir:

root@31a71014ad95:/usr/local/app# du -hs ./*|grep CRISP
12G     ./CRISPRCasFinder

I saw ps -s, inspect, system df as a recomended ways to estimate container disk usage stats, but has no success. All of them shows not actual values. So seems i'am using those tools in a wrong way i interpret numbers in a worng way.

From the mother host stats look like :

1. docker inspect gives https://pastebin.com/53vLji1p

2. docker ps -s gives

%docker container ps -sa
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS                        PORTS                      NAMES               SIZE
31a71014ad95        nvm:nvm_tag   "/bin/bash"              6 hours ago         Exited (137) 23 minutes ago                              cr                  3.52GB (virtual 5.44GB)
<...>

3. docker system df gives

%docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              8                   6                   5.52GB              3.021GB (54%)
Containers          6                   0                   3.541GB             3.541GB (100%)
Local Volumes       33                  6                   2.751GB             2.381GB (86%)
Build Cache         0                   0                   0B                  0B

None of those values look like container usage with du (shown above). Are those values are the things i should look for?

The most usefull and detailed exmple I found is https://www.projectatomic.io/blog/2016/03/daemon_option_basedevicesize/

But, it seems that, at my current docker version output is no more contains such fields as author uses for illustrations - no "base device size", no "DeviceSize"

So questions are:

1. how can i see the real container usage stats so i can then apply some tuning.

2. where is configuration of max container disk size placed?

Thank you!

UPDATE

I tried to restart docker app and it all ended in fatal space error - https://pastebin.com/twM3qN8N

So docker is not starting at all at the moment. Ok... I also found something very similar as an issue on gh - https://github.com/docker/for-mac/issues/3529

I saw before issues about misbehave, where deleting of qcow2 file was a fix, but always there really was a space issue where used and configured sizes where close, and the problem was in resizing. But seems this issue is about similar probem as my, cause issue author qcow2 file size is seems to be big enough.

I see that temp workaround is clear all data, but hope to find alternative, or at least some kind of explanation about disk usage metrics before i go brute clear.

UPDATE 2

partial decision without qcow clear but no understanding

Was keep on reading all kinds if articles and after reading http://phutchins.com/blog/2017/01/04/fixing-docker-no-space-left-on-device/

And noticed that strange 20Gb limitation, default in 2017. And thought. Lets imagine, that despite df -h inside container and docker desktop ui settings reports about 64Gb, i still have 20Gb limitation. Looks possible?

And installed qemu and added +5Gb to the image as was written in the article:

%qemu-img resize ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 +5G

And voila everything started and worked. I still see the same size for qemu file:

%ls -lah ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
-rw-r--r--@ 1 usss  staff    21G May 27 00:46 /Users/usss/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2

but usage form the container point of view changed a bit

root@31a71014ad95:/usr/local/app# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          68G   21G   45G  32% /
tmpfs            64M     0   64M   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
osxfs           234G  178G   55G  77% /shared_from_host
/dev/sda1        68G   21G   45G  32% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           2.0G     0  2.0G   0% /proc/acpi
tmpfs           2.0G     0  2.0G   0% /sys/firmware

it now states avail 45Gb(above there were 40Gb).

But at least everything works now, but absolutely no understanding where is shadowed consumption hides.

Answer 1

Disk space with desktop versions of docker can be limited by the disk you give to the embedded docker VM. Inside of this VM are the images, containers, named volumes, and any other files not explicitly mounted into the VM from an external source (e.g. the /Users directory on MacOS). You can tune this VM's settings from the docker preferences:

_{Image Credit: docker documentation}

More details are available from: https://docs.docker.com/docker-for-mac/space/

---

One important note on disk usage, you can run out of bytes of disk, and you can also run out of inodes. Both will give the same error. Use the df -i option to check indoes, e.g.:

df -h # to see disk free
df -hi # to see inodes free

---

If your error occurs while a container is running, and your container is then deleted, it's possible that the deletion of the container frees up that disk space.