Blog
Build logs and write-ups — what I built, what broke, and what it took to fix.
Progress on the homelab cluster over the last ten days: InfluxDB and the document archive moved in, the old Compose host runs nothing, the node disks grew from 20 to 50 GiB the hard way, and a few things moved out again.
Moving a MEGAcmd sync to a new host turned every file that had changed since its last upload into a conflict. On top of that, half the account quota was old versions and the rubbish bin, the package installs its own apt source, and syncs are two-way only.
A correction to my NAS maintenance notes: when a parent Kustomization applies the child, the next sync clears a manual suspend. Annotating the one object with reconcile=disabled holds, but it also stops Flux from pruning it.
The kubelet evicted pods because the node disks were tiny, and the cure was obviously bigger disks. Getting there took a trim of the thin pool, three reboots, and a night in which Home Assistant was down on and off for ninety minutes and etcd timed out because of one shared NVMe.
Taking the Talos collection from a green test cluster to the live one: the drift a read-only check showed, three bugs only production could find, and a feature I moved and then deleted because Flux already did it.
Applying a new machine config to a three-node cluster rebooted each node in turn. One came back on a different IP, because its DHCP reservation had been wrong for months and the node had never rebooted to notice.
I wanted real-cluster tests for an Ansible collection that drives Talos. On my machine neither Docker nor QEMU would give me a node in maintenance mode, so I tested the rendering instead: fixtures, talosctl validate, determinism and negative cases. What it caught, and the five CI runs it took to go green.
DNS lookups to the Pi-hole address succeeded, but its query log stayed empty. The answers came from the router, whose ad-block feature redirects every port-53 packet on the LAN to its own resolver. How I found out, and the one-line test that would have saved me the afternoon.
A Prometheus volume looked almost full while retentionSize was set. The real story: a single corrupt WAL segment had stopped every checkpoint for weeks, the WAL grew to 48.7 GiB, and the size cap spent the difference by deleting old blocks.
A follow-up on the cluster whose etcd shared one consumer drive with every storage replica. Flush latency fell from 23 ms to 3 ms, a filtered copy shrank a 9.9 GB SQLite database to 3.8 GB in ten minutes, and turning on discard cost more in replica rebuilds than in reboots.
A scale-down that never happened led me to leader elections lost on every node, and from there to a single drive holding every etcd member and every storage replica. Three replicas on one disk mostly just triple the writes.
Kubernetes has no notion of an active/passive Service. MetalLB in L2 mode with externalTrafficPolicy: Local is one, and I had been running it with a single replica the whole time.
The backup command I was asked to schedule did nothing on the current version: it printed usage text, exited 0 and wrote no file. So the job now checks the file it produces and refuses to upload anything that is not a real archive.
I added local semantic search to a small knowledge base, and the default LLM reranker took 33 seconds to put the wrong document first. Every cheaper path got it right, which I only found out by asking questions I already knew the answers to.
Retiring an unpatched WordPress on PHP 5.6 by replacing it with an Astro build pushed from a self-hosted Gitea runner to the same shared hosting.