Kubernetes
Clusters, workloads, storage and the ways they fail.
Series
Three etcd members, one NVMe
A consumer NVMe under every etcd member and storage replica, and what fixing it bought.
Posts
Progress on the homelab cluster over the last ten days: InfluxDB and the document archive moved in, the old Compose host runs nothing, the node disks grew from 20 to 50 GiB the hard way, and a few things moved out again.
The kubelet evicted pods because the node disks were tiny, and the cure was obviously bigger disks. Getting there took a trim of the thin pool, three reboots, and a night in which Home Assistant was down on and off for ninety minutes and etcd timed out because of one shared NVMe.
Applying a new machine config to a three-node cluster rebooted each node in turn. One came back on a different IP, because its DHCP reservation had been wrong for months and the node had never rebooted to notice.
A Prometheus volume looked almost full while retentionSize was set. The real story: a single corrupt WAL segment had stopped every checkpoint for weeks, the WAL grew to 48.7 GiB, and the size cap spent the difference by deleting old blocks.
A follow-up on the cluster whose etcd shared one consumer drive with every storage replica. Flush latency fell from 23 ms to 3 ms, a filtered copy shrank a 9.9 GB SQLite database to 3.8 GB in ten minutes, and turning on discard cost more in replica rebuilds than in reboots.
A scale-down that never happened led me to leader elections lost on every node, and from there to a single drive holding every etcd member and every storage replica. Three replicas on one disk mostly just triple the writes.
Kubernetes has no notion of an active/passive Service. MetalLB in L2 mode with externalTrafficPolicy: Local is one, and I had been running it with a single replica the whole time.
The backup command I was asked to schedule did nothing on the current version: it printed usage text, exited 0 and wrote no file. So the job now checks the file it produces and refuses to upload anything that is not a real archive.
Notes
Why open() on a passed-through device node returns EPERM as root, how the cgroup v2 eBPF device filter causes it, and the short control test that tells you it is the cause.
Pod Security Admission enforces on Pods, not on workload templates. A privileged DaemonSet is created, reports healthy and never runs a pod, and a server-side dry run of it will not tell you.
mDNS and SSDP do not cross the pod network. A macvlan second interface through Multus gets around that, provided the node NIC, the CNI plugins and the attachment routes are all right. Each of them fails silently when it is not.
Longhorn always mounts an NFS backup target as nfs4, including in its fallback chain, so a NAS that only serves NFSv3 cannot be a backup target. How to check the server, and what enabling v4 on an older QNAP involves.
The default service range 10.96.0.0/12 reaches 10.111.255.255. If the node network sits inside it, Talos treats the node address as Kubernetes-only and leaves it out of the API certificate and etcd. The symptoms, how to see it with talosctl, and the smaller service range that fixes it.
From 1.14 Talos moves kubelet, labels, taints and the service range out of the v1alpha1 config into typed documents, and rejects the old fields next to them. A node on an older version rejects the new documents. What that does to patches, fixtures and upgrade order.
Talos labels control-plane nodes as excluded from external load balancers, and MetalLB speakers honour the label. On a cluster without workers that leaves no node to announce a layer 2 address. How the speaker log and servicel2status show it, and the patch that removes the label.
Without --iface, flannel records the address of whichever interface it found first as the node's VXLAN endpoint. Change that address and cross-node traffic dies while every node still looks healthy. The annotation to check and the restart that fixes it.
Talos binds kube-controller-manager, kube-scheduler and kube-proxy metrics to the loopback address, so kube-prometheus-stack shows nine dead targets. The machine-config patch, why kube-proxy needs a different treatment, and why talosctl upgrade-k8s is not the small operation it sounds like.
A macvlan parent interface left on DHCP installs a LAN default route at the same metric as the real one. Egress drifts onto the wrong network and replies to LAN clients take a different path than their requests. Why a /32 with no gateway is the fix, and the commands that show the problem.
Sixty evicted pods in an hour on a node with an 18 GiB disk. The kubelet's 15% imagefs threshold leaves about 1.4 GiB of headroom there, so one burst of image pulls evicts everything. How to read the numbers, clean up, and what pruning does and does not buy.
A Deployment from a Talos extraManifests entry was missing for at least a month and nothing noticed. How to tell that leftover objects came from Talos rather than GitOps, how to date a disappearance with kube-state-metrics, and what a missing kubelet serving-cert approver eventually costs.
The stock etcdHighFsyncDurations warning fired about 200 times in a week on a homelab. Why histogram bucket edges decide what a threshold means, how kube-prometheus-stack lets you change it, and why raising it hides a real disk problem.
I put an MQTT web client in front of Mosquitto and tried to keep the WebSocket port cluster-only. bind_interface does nothing for websockets listeners in 2.0.22, an interface name as host hangs the broker, and a pod IP rendered by an init container is what works.
Semaphore panicked on a bad Port value because Kubernetes injected SEMAPHORE_PORT=tcp://... for a Service with the same name. Turn off enableServiceLinks when the Service name matches the app, plus two migration gotchas from moving a config directory with a huge git checkout.
How to feed Homepage widget passwords from a SOPS Secret with HOMEPAGE_VAR placeholders, why annotation widgets cannot use them, and why rewriting a public repo's history does not undo a leak.
After a reboot Flux reported etcd timeouts and etcd warned about a slow disk. Talos has no per-process I/O view, but /proc/<pid>/io can be read through talosctl; here it pointed at a Longhorn snapshot purge.
FTLCONF_LOCAL_IPV4 does nothing in Pi-hole v6, and the "Primary IP addresses" panel is detection-only. The setting that controls what pi.hole resolves to is dns.reply.host.IPv4.
Some images start as root, chown their data directory and then switch user. runAsNonRoot breaks them. Dropping all capabilities and adding back CHOWN, SETUID and SETGID is the tightest setup that still starts.
After Flux applies a rotated Secret, the pod keeps the old value in its environment until it restarts. A 302 from the app does not prove the new password works.
Translating a Compose file to a Deployment, command: in Kubernetes silently drops the image ENTRYPOINT. Gotenberg lost its init wrapper and Paperless' exporter lost its environment. Check what the old container actually ran.
A Service without a selector plus a hand-written EndpointSlice points Traefik at any address, and the existing wildcard certificate and DNS keep working. The risky part is the cutover order, because Flux prunes less than you think.
MetalLB only hands out addresses from its own pool, so a pod that has to own a LAN address (a scanner target, say) needs a macvlan interface. The hard part is the handover: two hosts answering ARP for one address break both.
After each rolling reboot, Deployment pods stayed behind in phase Succeeded. The pod garbage collector only acts above 12,500 terminated pods, and lowering that would also eat CronJob history. A small CronJob that deletes only orphans owned by ReplicaSets, StatefulSets and DaemonSets does the job.
On Talos, the chart's etcd Service selects no pods, so it looks like dead weight. Turning the whole kubeEtcd block off also drops the etcd alert rules and the Grafana dashboard. Turn off only the Service and ServiceMonitor.
A 20 GiB Longhorn volume took 36.7 GiB per replica. Home Assistant wrote its daily 1.6 GB backup to the same volume, and seven daily snapshots each kept one. Sending backups off the volume fixed the growth.
Homepage asks every server in docker.yaml on each /api/services call. One host that times out delays the whole response by minutes, and the page renders no tiles. Time the API from inside the pod to find it.
kubectl apply --dry-run=server checks the HelmRelease CRD schema and stops there. Rendering the chart is the only cheap way to catch a mistake inside spec.values.