I am maintaining multiple servers which are situated in very badly connected remote locations all over the world. These servers run multiple docker containers, some of which based on fairly big images (we have a separate on-going effort to reduce this, but this is more long term).
These servers are only reachable through reverse ssh tunnels and the network connections drop very frequently. The bandwidth is in general also fairly low (sometimes only ~50kbps or even less).
AFAIK, I have two options to transfer images, each with some issues.
Using a docker repository and docker pull:
- Does not deal well with network interruptions, the whole download restarts from scratch.
- Does not redownload already downloaded layers, reducing the time and bandwidth used to download newer versions with only changes in the upper layers.
Using docker save, rsync and docker load:
- Rsync allows to continue a partial download, we can start a download and have it retry untill finished.
- It's only possible to save and transfer the whole image, including all layers. So we have to download the complete image even if only a couple kB in the topmost layer changed compared to the previous version.
So, my question, is there any other way in which to transfer these images which combines the benefits of both approaches? So which can resume an interrupted download but does not require to download complete images every time?