2

I am maintaining multiple servers which are situated in very badly connected remote locations all over the world. These servers run multiple docker containers, some of which based on fairly big images (we have a separate on-going effort to reduce this, but this is more long term).

These servers are only reachable through reverse ssh tunnels and the network connections drop very frequently. The bandwidth is in general also fairly low (sometimes only ~50kbps or even less).

AFAIK, I have two options to transfer images, each with some issues.

Using a docker repository and docker pull:

  1. Does not deal well with network interruptions, the whole download restarts from scratch.
  2. Does not redownload already downloaded layers, reducing the time and bandwidth used to download newer versions with only changes in the upper layers.

Using docker save, rsync and docker load:

  1. Rsync allows to continue a partial download, we can start a download and have it retry untill finished.
  2. It's only possible to save and transfer the whole image, including all layers. So we have to download the complete image even if only a couple kB in the topmost layer changed compared to the previous version.

So, my question, is there any other way in which to transfer these images which combines the benefits of both approaches? So which can resume an interrupted download but does not require to download complete images every time?

  • Never underestimate the bandwidth of a station wagon full of tape drives going down the interstate at 55mph. In other words, have you looked into offline file transfer? For connections as slow as you note, and for large enough image files, it may be a worthwhile consideration until other factors improve. – music2myear Sep 6 '18 at 16:46
  • Well, conditions will not improve much, that's a given. Besides that, these servers are mostly located in remote, badly reachable locations without a lot of people coming and going. Many of these locations are also at least a full day of multi-means travel removed from where I am based. So even with the bad networking conditions that we have, the internet still seems to be the most efficient way to transfer data. – Ozymandias Sep 14 '18 at 15:57
  • There's things like ResinOS although I'm not sure how much that'd help since it's still just doing a regular Docker image transfer afaik but maybe with built in error handling – nijave Sep 16 '18 at 21:35
1

A few ideas come to mind

  1. If you're building images smartly, can you combine both techniques and use rsync for the initial load and docker pull for subsequent updates (since it will only pull changed layers at this point)

  2. You can tag more layers so instead of having 1 Docker image with 10 layers, you have 5 Docker images with 2 layers. You could either split into multiple Dockerfiles or tag layers after the fact ("docker tag layerHash tagName" for details https://gist.github.com/dasgoll/476ecc7a057ac885f0be)

  3. Not positive on this one, but you should be able to export single layers instead of entire images by specifying the layer hash. The only difference between layers and "images" are images have human-readable tags assigned to the layer

  4. Untar the exported image, transfer only new layers, and re-tar by combining new layers with cached layers (when you export an image it's just a tar with 1 tar per layer inside)

  5. Don't use Docker for building--just assemble and run. With this one, you'd just transfer a rootfs over (possibly as regular files and let rsync handle compression) then tar them and do a plain Dockerfile which just does ADD rootfs.tar / and has your run command. This would ensure nothing ever gets copied more than necessary

You could also combine these approaches with something like Bittorrent to distribute the files

  • Well, I looked at some of these options, but docker is very picky about reusing layers, see this open issue: github.com/moby/moby/issues/23684. Some of your suggestions are interesting and I didn't think of them yet though, so I'll give it a try anyway, thanks! – Ozymandias Sep 14 '18 at 15:59
  • According to latest comments to this issue, it's still not fixed (canceling a pull causes even fully downloaded layers to be discarded): github.com/moby/moby/issues/26146 The follow-up issue is still a work in progress too: github.com/moby/moby/pull/38738 – V-R Nov 11 '20 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.