Consider these two commands run against my cluster.

First roll up all known used ram as reported by top nodes:

$ kubectl --no-headers=true top nodes | awk 'BEGIN { total=0 } { total+=$4 } END { print total, "Mi" }'
504972 Mi

Then by all pods in top pods

phs@anvil:~$ kubectl --no-headers=true --all-namespaces=true top pods |  awk 'BEGIN { total=0 } { total+=$4 } END { print total, "Mi" }'
163503 Mi

I've already confirmed that all living pods appear in top pods.

What is with the huge discrepancy? Where does the extra ram go?


It appears the difference comes down to buffers. If we run free -mw on each node I can see significant (10-15 GB) amounts of ram sunk there.

$ kubectl get nodes -o name | cut -d / -f 2 | while read node; do ssh -n admin@$node 'free -mw | fgrep Mem:'; done | awk 'BEGIN { total=0 } { total+=$6 } END { print total, "Mi" }'
278815 Mi

Their total is approximately the missing mount (these commands were run at different times, and so exact equality shouldn't be expected.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.