How to limit Google Compute Engine CPU utilization to 100%

Question

I'm running batch jobs on n1-highcpu-32 Ubuntu 14.04 instances, which have 16 physical (32 virtualized) cores. This should be a trivial case (compared to MPI), but the CPU utilization is all over the place, as reported by top.

Some of the jobs run at 300%+ CPU utilization, while others run at only 10%. Therefore, the time to complete the entire batch is 10x longer than it should be. Each of the jobs is running the exact same code, which is not multi-threaded.

Is there a way to prevent this behavior? Perhaps limit CPU utilization to 100% or something?

Answer 1

You can run each process in a separate Docker container. By default, --cpu-quota is set to 0, which provides 100% of a single CPU, so you should be all set. If you want to choose the value manually, use:

docker run -it --cpu-quota="..." [container] [command] [args]

Alternatively, you can assign specific CPUs to a given Docker process via --cpu-setcpus flag, e.g.:

docker run -it --cpuset-cpus="1,3" [container] [command] [args]

For details, see the docker run docs:

CPU quota constraint

The --cpu-quota flag limits the container’s CPU usage. The default 0 value allows the container to take 100% of a CPU resource (1 CPU). The CFS (Completely Fair Scheduler) handles resource allocation for executing processes and is default Linux Scheduler used by the kernel. Set this value to 50000 to limit the container to 50% of a CPU resource. For multiple CPUs, adjust the --cpu-quota as necessary. For more information, see the CFS documentation on bandwidth limiting.