Blog
Build logs and write-ups — what I built, what broke, and what it took to fix.
I wanted real-cluster tests for an Ansible collection that drives Talos. On my machine neither Docker nor QEMU would give me a node in maintenance mode, so I tested the rendering instead: fixtures, talosctl validate, determinism and negative cases. What it caught, and the five CI runs it took to go green.
DNS lookups to the Pi-hole address succeeded, but its query log stayed empty. The answers came from the router, whose ad-block feature redirects every port-53 packet on the LAN to its own resolver. How I found out, and the one-line test that would have saved me the afternoon.
A Prometheus volume looked almost full while retentionSize was set. The real story: a single corrupt WAL segment had stopped every checkpoint for weeks, the WAL grew to 48.7 GiB, and the size cap spent the difference by deleting old blocks.
A follow-up on the cluster whose etcd shared one consumer drive with every storage replica. Flush latency fell from 23 ms to 3 ms, a filtered copy shrank a 9.9 GB SQLite database to 3.8 GB in ten minutes, and turning on discard cost more in replica rebuilds than in reboots.
A scale-down that never happened led me to leader elections lost on every node, and from there to a single drive holding every etcd member and every storage replica. Three replicas on one disk mostly just triple the writes.
Kubernetes has no notion of an active/passive Service. MetalLB in L2 mode with externalTrafficPolicy: Local is one, and I had been running it with a single replica the whole time.
The backup command I was asked to schedule did nothing on the current version: it printed usage text, exited 0 and wrote no file. So the job now checks the file it produces and refuses to upload anything that is not a real archive.
I added local semantic search to a small knowledge base, and the default LLM reranker took 33 seconds to put the wrong document first. Every cheaper path got it right, which I only found out by asking questions I already knew the answers to.
Retiring an unpatched WordPress on PHP 5.6 by replacing it with an Astro build pushed from a self-hosted Gitea runner to the same shared hosting.