1

I am using continuumio/miniconda3:latest as the base image for my container. It looks to be debian based. I run it under Amazon Linux 2. I would like to optimally use compressed memory, perhaps zram, in the miniconda3 container in privileged mode. The container has access to 4 to 128 vCPUs and 6 GiB to 3.7 TiB of RAM, depending upon the job at hand. For what it's worth, the container doesn't seem to currently have systemd. My motivation is that the container has a large amount of unused CPU resources but insufficient memory for a big data job. Ideally I would want 2:1 or if possible 3:1 the virtual memory as the physical memory.

For what it's worth, the container has a /proc filesystem, and it also has privileged mode access in case it's relevant. The host has a /proc filesystem that has been mounted into the container at /host/proc.

What is the best way for me to update my docker image and container to optimally configure compressed memory using all vCPUs, preferably without hardcoding the amount? In principle I'd like to execute an entrypoint script, perhaps to configure and enable zram.

Improve this question
4
  • 1
    I believe memory compression is a kernel feature, not a Docker feature. How much real memory do you have, and what problems are you having?
    – harrymc
    Feb 12, 2021 at 9:25
  • @harrymc Regarding it being a kernel feature, I have now added a paragraph with info about the /proc filesystem. The real memory is about 3.7 TiB and there are 128 vCPUs available for de/compression. The problem is that pandas is running out of memory despite valiant attempts to minimize its memory usage. In general though, using compressed memory would be great because it would result in significant cost savings for many memory-bound jobs even when the container is not running out of memory.
    – Asclepius
    Feb 12, 2021 at 15:36
  • 2
    You are serious about physical memory of 3.7 TiB and 128 physical CPUs? This would make it a supercomputer. And that's not enough for one program?
    – harrymc
    Feb 13, 2021 at 8:49
  • @harrymc It appears that due to various Pandas bugs, indeed the memory has proven insufficient. I will however switch to another tool.
    – Asclepius
    Feb 18, 2021 at 19:17

1 Answer 1

Reset to default
1

Memory compression is a kernel feature, not a Docker feature. In most operating systems, this feature is enabled automatically.

You cannot configure compressed memory in the Dockerfile installation.

If you stated your real problem and the real physical data about your computer, we might be able to help.

Improve this answer
6
  • 1
    I believe this answer in its current form to be less than useful because the Dockefile can in principle copy a script into the image which runs when the container is started to configure zram. The question has been updated to clarify this goal.
    – Asclepius
    Feb 18, 2021 at 19:14
  • Docker is isolated from the host operating system and cannot allocate real zram.
    – harrymc
    Feb 18, 2021 at 19:45
  • As I have noted in the question, the container is running in privileged mode, with rw access to both /proc and also independently to the host's /proc. These permissions should be more than sufficient.
    – Asclepius
    Feb 18, 2021 at 20:10
  • Why do you think that access to /proc is enough to modify kernel parameters?
    – harrymc
    Feb 18, 2021 at 20:53
  • What is required? /sys can be similarly accessed and mounted. Let's find a way to get this working instead of finding reasons for it to not work.
    – Asclepius
    Feb 18, 2021 at 23:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .