Kubernetes
Clusters, workloads, storage and the ways they fail.
Series
Three etcd members, one NVMe
A consumer NVMe under every etcd member and storage replica, and what fixing it bought.
Posts
A Prometheus volume looked almost full while retentionSize was set. The real story: a single corrupt WAL segment had stopped every checkpoint for weeks, the WAL grew to 48.7 GiB, and the size cap spent the difference by deleting old blocks.
A follow-up on the cluster whose etcd shared one consumer drive with every storage replica. Flush latency fell from 23 ms to 3 ms, a filtered copy shrank a 9.9 GB SQLite database to 3.8 GB in ten minutes, and turning on discard cost more in replica rebuilds than in reboots.
A scale-down that never happened led me to leader elections lost on every node, and from there to a single drive holding every etcd member and every storage replica. Three replicas on one disk mostly just triple the writes.
Kubernetes has no notion of an active/passive Service. MetalLB in L2 mode with externalTrafficPolicy: Local is one, and I had been running it with a single replica the whole time.
The backup command I was asked to schedule did nothing on the current version: it printed usage text, exited 0 and wrote no file. So the job now checks the file it produces and refuses to upload anything that is not a real archive.
Notes
Why open() on a passed-through device node returns EPERM as root, how the cgroup v2 eBPF device filter causes it, and the short control test that tells you it is the cause.
Pod Security Admission enforces on Pods, not on workload templates. A privileged DaemonSet is created, reports healthy and never runs a pod, and a server-side dry run of it will not tell you.
mDNS and SSDP do not cross the pod network. A macvlan second interface through Multus gets around that, provided the node NIC, the CNI plugins and the attachment routes are all right. Each of them fails silently when it is not.
Longhorn always mounts an NFS backup target as nfs4, including in its fallback chain, so a NAS that only serves NFSv3 cannot be a backup target. How to check the server, and what enabling v4 on an older QNAP involves.
The default service range 10.96.0.0/12 reaches 10.111.255.255. If the node network sits inside it, Talos treats the node address as Kubernetes-only and leaves it out of the API certificate and etcd. The symptoms, how to see it with talosctl, and the smaller service range that fixes it.
From 1.14 Talos moves kubelet, labels, taints and the service range out of the v1alpha1 config into typed documents, and rejects the old fields next to them. A node on an older version rejects the new documents. What that does to patches, fixtures and upgrade order.
Talos labels control-plane nodes as excluded from external load balancers, and MetalLB speakers honour the label. On a cluster without workers that leaves no node to announce a layer 2 address. How the speaker log and servicel2status show it, and the patch that removes the label.
Without --iface, flannel records the address of whichever interface it found first as the node's VXLAN endpoint. Change that address and cross-node traffic dies while every node still looks healthy. The annotation to check and the restart that fixes it.
Talos binds kube-controller-manager, kube-scheduler and kube-proxy metrics to the loopback address, so kube-prometheus-stack shows nine dead targets. The machine-config patch, why kube-proxy needs a different treatment, and why talosctl upgrade-k8s is not the small operation it sounds like.
A macvlan parent interface left on DHCP installs a LAN default route at the same metric as the real one. Egress drifts onto the wrong network and replies to LAN clients take a different path than their requests. Why a /32 with no gateway is the fix, and the commands that show the problem.
Sixty evicted pods in an hour on a node with an 18 GiB disk. The kubelet's 15% imagefs threshold leaves about 1.4 GiB of headroom there, so one burst of image pulls evicts everything. How to read the numbers, clean up, and what pruning does and does not buy.
A Deployment from a Talos extraManifests entry was missing for at least a month and nothing noticed. How to tell that leftover objects came from Talos rather than GitOps, how to date a disappearance with kube-state-metrics, and what a missing kubelet serving-cert approver eventually costs.
The stock etcdHighFsyncDurations warning fired about 200 times in a week on a homelab. Why histogram bucket edges decide what a threshold means, how kube-prometheus-stack lets you change it, and why raising it hides a real disk problem.
I put an MQTT web client in front of Mosquitto and tried to keep the WebSocket port cluster-only. bind_interface does nothing for websockets listeners in 2.0.22, an interface name as host hangs the broker, and a pod IP rendered by an init container is what works.
Semaphore panicked on a bad Port value because Kubernetes injected SEMAPHORE_PORT=tcp://... for a Service with the same name. Turn off enableServiceLinks when the Service name matches the app, plus two migration gotchas from moving a config directory with a huge git checkout.
How to feed Homepage widget passwords from a SOPS Secret with HOMEPAGE_VAR placeholders, why annotation widgets cannot use them, and why rewriting a public repo's history does not undo a leak.
kubectl apply --dry-run=server checks the HelmRelease CRD schema and stops there. Rendering the chart is the only cheap way to catch a mistake inside spec.values.