How to prevent high CPU usage from shutting down system?

Question

I work on a Linux system running Java JVM's that create new threads (processes) to handle the work load. Thread complexity can't be predicted beforehand and some threads end up doing more CPU intensive work than other threads We've run into a problem when so many threads are started that the CPU usage gets uncomfortably close to 100% with low percentage of idle. I understand if the CPU usage gets too high, other running services are impacted, for example the system won't accept new ssh logins until it it rebooted.

Isn't there an automated process monitor that can be configured at the system level to prevent CPU usage of user processes from getting too high, so that system services won't be impacted?

Answer 1

Most user processes under Linux have the same priority level when started and share time-slices equally. All threads also share equally in time slices, so when there are too many of them, some other process may need to wait a long while for its next time-slice.

There are at least three ways to control how much CPU time a process gets:

• The nice command can manually lower the task's priority.
• The cpulimit command can pause a process to avoid it exceeding a limit limit.
• Linux control groups (cgroups) can limit the amount of resources available to the process.

For more information see the article
Restricting process CPU usage using nice, cpulimit, and cgroups.

Answer 2

Alternate solution: if you run your JVM as a Docker container, the docker run command has many options to limit CPU usage (number of CPUs, CPU quota...).

Using containers also makes your application easier to deploy and manage.