I grew the node disks to stop the evictions, and the rollout hurt more than the problem
The kubelet evicted pods because the node disks were tiny, and the cure was obviously bigger disks. Getting there took a trim of the thin pool, three reboots, and a night in which Home Assistant was down on and off for ninety minutes and etcd timed out because of one shared NVMe.
A while ago I wrote about the kubelet evicting everything on a node with an 18 GiB disk, and ended with “the lasting answer is a bigger disk”. I knew that and didn’t do it. This is what happened when I finally did, on the day the problem bit me again.
The merge that started it
I had a Paperless stack ready to move into the cluster, with a large image and a handful of helper containers. I merged it and Flux deployed it. Within minutes every node reported DiskPressure, the kubelets evicted about a dozen pods, and Home Assistant was among them. It was down for roughly twelve minutes, because it is pinned to one node by its volume and the disk-pressure taint kept its replacement from starting. DiskPressure only clears five minutes after the condition stops, even when the free space is already back.
I suspended the new stack, scaled it to zero and waited. Everything recovered within about fifteen minutes. The cause wasn’t mysterious: my own notes already said these nodes have about 1.4 GiB of headroom. I had searched them for Paperless and not for how much room the nodes had. Before deploying anything with a big image, read the node’s free space from the kubelet’s summary and compare it to what you’re about to pull, simultaneous pulls included.
Trim first, then grow
The disks lived in OpenTofu, 20 GiB for the OS and 250 GiB for Longhorn, all on one thin pool on the Proxmox host. That pool was 85 percent allocated, and Longhorn only used about 223 GiB of the 750 GiB of data disks. The rest was blocks the guests had freed and never returned, because the trim I had planned weeks earlier had never run.
So I ran it, from my workstation: Longhorn’s own filesystem trim on each of the
19 attached volumes, then fstrim on each node’s data and system mounts from a
privileged pod. The Longhorn trims alone moved nothing. The pool stayed at 85.46
percent until the node-level fstrim ran, and then it went to 68, 46 and 25
percent, node by node. You need both layers. One more thing I learned: the port
forward for the Longhorn API has to go to the frontend service. The backend
service refuses connections from a forward, because the manager doesn’t listen
on the pod’s loopback.
The trim doesn’t give the kubelet any room, by the way. The nodes’ free space was the same 3 to 4.5 GB afterwards. It only made it safe to grow.
Growing, which was the easy part
The new size was 50 GiB, applied online with qm resize. I recorded it in the
state with a refresh-only apply, because a normal one would also have planned to
destroy an unrelated image download, which I’d hit before.
Talos grows the ephemeral partition by itself, but only when the node reboots.
I had a health-gated reboot script: etcd ready, nodes Ready, no DiskPressure, Longhorn volumes healthy. For the first node I cordoned and drained it. That was the mistake. Draining moved 33 pods onto the two nodes that still had small disks, they pulled images there, and DiskPressure came back four times, taking Home Assistant down with it each time. The other two nodes I rebooted in place, which restarts the pods from images already on the node. Rebooting in place would have been the better plan from the start. The node with the USB dongle Home Assistant needs was also the node whose reboot hid the volume driver for ten minutes afterwards.
The night the NVMe lost
By about one in the morning the disks were big, and Paperless could finally start. It started pulling its images just as Longhorn was resyncing three volumes (Prometheus at 47 GB, InfluxDB at 16 GB, Home Assistant’s config) after the reboots. API calls took 9 to 30 seconds. etcd’s raft term counter went from 1087 to 1110 in that stretch, so leadership kept moving. Kubelets logged “Failed to update lease: etcdserver: request timed out”, and the API address stopped answering at all.
At that point I measured instead of guessing. On the Proxmox host vmstat showed
25 percent iowait and a load average of 51. The virtual machines weren’t pushing
much, around 70 MB/s of reads and 17 MB/s of writes, but the device’s write
latency was about 245 ms against 26 ms for reads, while SMART was perfectly
clean. One consumer NVMe carries every VM, Longhorn replica and etcd member, and
sync writes on it stall when enough things want it at once. I had written about
etcd on a single NVMe before and
this was the same disk showing the same weakness.
Part of the load was me. I had three monitors polling the API every 15 to 45 seconds, which during an etcd-latency episode just adds to it. I stopped them, turned Longhorn’s concurrent rebuild limit per node from 5 to 1, and deleted the Paperless pods. Load fell from 51 to about 4 over fifteen minutes. A resync of the 47 GB volume still took around 40 minutes at the lower limit.
What it bought
The nodes now have 29 to 36 GB free for the kubelet instead of 3 to 4.5, and the Paperless stack runs. Nothing was lost, but Home Assistant was down on and off for about ninety minutes, and etcd timed out for several minutes on two occasions.
What I’d do differently next time: reboot in place and don’t drain onto small disks, throttle Longhorn rebuilds before the first reboot and put the limit back afterwards, run one light health check every couple of minutes instead of three loops, and never overlap a rebuild with a big image pull. When the API address stops answering, talk to each node’s address directly (the leader’s own API server is the slowest one).
Two smaller things I found on the way. The root Flux Kustomization un-suspends a child I had suspended by hand within about ten minutes, so a manual suspend is not a durable way to park something. And when the controller manager falls behind during etcd timeouts, scaling a Deployment to zero can leave its pods running for minutes; deleting them directly worked.