Total memory usage as reported by kubectl top nodes vs top pods

Question

Consider these two commands run against my cluster.

First roll up all known used ram as reported by top nodes:

$ kubectl --no-headers=true top nodes | awk 'BEGIN { total=0 } { total+=$4 } END { print total, "Mi" }'
504972 Mi

Then by all pods in top pods

phs@anvil:~$ kubectl --no-headers=true --all-namespaces=true top pods |  awk 'BEGIN { total=0 } { total+=$4 } END { print total, "Mi" }'
163503 Mi

I've already confirmed that all living pods appear in top pods.

What is with the huge discrepancy? Where does the extra ram go?

Answer 1

It appears the difference comes down to buffers. If we run free -mw on each node I can see significant (10-15 GB) amounts of ram sunk there.

$ kubectl get nodes -o name | cut -d / -f 2 | while read node; do ssh -n admin@$node 'free -mw | fgrep Mem:'; done | awk 'BEGIN { total=0 } { total+=$6 } END { print total, "Mi" }'
278815 Mi

Their total is approximately the missing mount (these commands were run at different times, and so exact equality shouldn't be expected.)