Kubelet evicts on image headroom, and small node disks have almost none

Sixty evicted pods in an hour on a node with an 18 GiB disk. The kubelet's 15% imagefs threshold leaves about 1.4 GiB of headroom there, so one burst of image pulls evicts everything. How to read the numbers, clean up, and what pruning does and does not buy.

After some network work on my cluster, a Gitea pod sat in Init:0/3 with a Longhorn Multi-Attach error, and two namespaces were full of Evicted pods: sixty of them, all within one hour, all on the same node.

The Multi-Attach error was the easy half. An RWO volume can’t attach to a new pod while the old one still holds it; it cleared by itself once the old pod terminated.

Why sixty pods at once

The evicted pods said The node had condition: [DiskPressure], and a few said low on resource: ephemeral-storage. The kubelet’s thresholds explain it. These are the defaults, and you can read yours from the node’s configz:

kubectl get --raw /api/v1/nodes/<node>/proxy/configz | jq .kubeletconfig.evictionHard
# "imagefs.available": "15%", "nodefs.available": "10%"

My Talos VMs have 17.8 GiB disks, roughly 10 to 11 GiB of which is container images. Fifteen percent of 17.8 GiB is 2.67 GiB, so the kubelet starts evicting when less than that is free, and the node had about 4.1 GiB free to begin with. That’s around 1.4 GiB of headroom. A burst of rescheduling and image pulls (a new Gitea image, a debug image, pods moving nodes) crosses it in minutes, and then the kubelet rejects every pod scheduled to the node, not just the greedy one.

Reading the numbers

talosctl has no df, and talosctl usage /var is painfully slow. The kubelet’s own summary is quick:

kubectl get --raw /api/v1/nodes/<node>/proxy/stats/summary | jq '.node.fs, .node.runtime.imageFs'

Cleaning up

Evicted pods remain as Failed objects until someone deletes them. They’re harmless and noisy:

kubectl delete pods -A --field-selector=status.phase=Failed

For images, I listed what the runtime had cached on each node, compared digests with the imageID of every container actually running there, and removed the rest with talosctl image remove --namespace cri <ref>. That bought one to three GiB per node. It also isn’t permanent: the cache fills up again with images of pods that later move, and image garbage collection only starts at 85% usage, so old versions (I had two Longhorn engine releases cached) pile up until then.

Pruning gives you breathing room. The lasting answer is a bigger disk, because with this little room the next change that moves a few pods will do it again.