Over the past few months, I've had an extremely irritating problem with my Linux system: it stutters on Firefox audio playback, mouse movement, etc., with a tiny sub-second (but still noticeable) stutter every few seconds. The issue worsens as the memory cache fills, or when I have highly disk/memory intensive programs running (e.g. backup software
restic). However, when the cache isn't full (e.g. under very light load), everything runs very smoothly.
perf top output, I see that
list_lru_count_one has high overhead (~20%) during these periods of lag.
htop also shows
kswapd0 using 50-90% CPU (though it feels like the impact is much greater than that). During times of extreme lag, the
htop CPU meter is often dominated by kernel CPU usage.
The only workaround I've found is either force the kernel to keep free memory (
sysctl -w vm.min_free_kbytes=1024000) or to continuously drop memory caches through
echo 3 > /proc/sys/vm/drop_caches. Neither is ideal, of course, and neither completely solves the stuttering either; it only makes it less frequent.
Does anyone have any ideas on why this might be occurring?
- i7-4820k with 20 GB of (mismatched) DDR3 RAM
- Reproduced on Linux 4.14-4.18 on NixOS unstable
- Runs Docker containers and Kubernetes in the background (which I feel like shouldn't create microstuttering?)
What I've already tried
- Changing I/O schedulers (bfq), using multiqueue I/O schedulers
- Using the
-ckpatchset by Con Kolivas (didn't help)
- Disabling swap, changing swappiness, using zram
EDIT: For clarity, here's a picture of
perf during such a lag spike. Note the high
list_lru_count_one CPU load and the
kswapd0 + high kernel CPU usage.