Build logs, notes and project write-ups on infrastructure automation, Kubernetes, GitOps and the homelab they run on.https://cs-ol.de/https://cs-ol.de/blog/testing-a-talos-collection-without-a-cluster/https://cs-ol.de/blog/testing-a-talos-collection-without-a-cluster/I wanted real-cluster tests for an Ansible collection that drives Talos. On my machine neither Docker nor QEMU would give me a node in maintenance mode, so I tested the rendering instead: fixtures, talosctl validate, determinism and negative cases. What it caught, and the five CI runs it took to go green.Sat, 03 Oct 2026 00:00:00 GMTtestingcigiteahttps://cs-ol.de/blog/pihole-answered-every-query-and-logged-none/https://cs-ol.de/blog/pihole-answered-every-query-and-logged-none/DNS lookups to the Pi-hole address succeeded, but its query log stayed empty. The answers came from the router, whose ad-block feature redirects every port-53 packet on the LAN to its own resolver. How I found out, and the one-line test that would have saved me the afternoon.Fri, 02 Oct 2026 00:00:00 GMTdnspi-holewslmetallbhttps://cs-ol.de/blog/one-corrupt-wal-segment-ate-prometheus-history/https://cs-ol.de/blog/one-corrupt-wal-segment-ate-prometheus-history/A Prometheus volume looked almost full while retentionSize was set. The real story: a single corrupt WAL segment had stopped every checkpoint for weeks, the WAL grew to 48.7 GiB, and the size cap spent the difference by deleting old blocks.Sat, 26 Sep 2026 00:00:00 GMTlonghornstoragemonitoringhttps://cs-ol.de/blog/after-the-nvme-what-the-fixes-bought/https://cs-ol.de/blog/after-the-nvme-what-the-fixes-bought/A follow-up on the cluster whose etcd shared one consumer drive with every storage replica. Flush latency fell from 23 ms to 3 ms, a filtered copy shrank a 9.9 GB SQLite database to 3.8 GB in ten minutes, and turning on discard cost more in replica rebuilds than in reboots.Fri, 25 Sep 2026 00:00:00 GMTetcdstoragelonghornproxmoxsqlitehttps://cs-ol.de/blog/three-etcd-members-one-consumer-nvme/https://cs-ol.de/blog/three-etcd-members-one-consumer-nvme/A scale-down that never happened led me to leader elections lost on every node, and from there to a single drive holding every etcd member and every storage replica. Three replicas on one disk mostly just triple the writes.Wed, 23 Sep 2026 00:00:00 GMTetcdstoragelonghornproxmoxhttps://cs-ol.de/blog/active-passive-failover-one-vip/https://cs-ol.de/blog/active-passive-failover-one-vip/Kubernetes has no notion of an active/passive Service. MetalLB in L2 mode with externalTrafficPolicy: Local is one, and I had been running it with a single replica the whole time.Mon, 21 Sep 2026 00:00:00 GMTmetallbdnshigh-availabilityhttps://cs-ol.de/blog/backup-job-that-backs-up-nothing/https://cs-ol.de/blog/backup-job-that-backs-up-nothing/The backup command I was asked to schedule did nothing on the current version: it printed usage text, exited 0 and wrote no file. So the job now checks the file it produces and refuses to upload anything that is not a real archive.Mon, 21 Sep 2026 00:00:00 GMTbackupspi-holesshhttps://cs-ol.de/blog/reranking-made-search-worse/https://cs-ol.de/blog/reranking-made-search-worse/I added local semantic search to a small knowledge base, and the default LLM reranker took 33 seconds to put the wrong document first. Every cheaper path got it right, which I only found out by asking questions I already knew the answers to.Sun, 13 Sep 2026 00:00:00 GMTsearchllmwslhttps://cs-ol.de/blog/static-site-on-shared-hosting/https://cs-ol.de/blog/static-site-on-shared-hosting/Retiring an unpatched WordPress on PHP 5.6 by replacing it with an Astro build pushed from a self-hosted Gitea runner to the same shared hosting.Sat, 12 Sep 2026 00:00:00 GMTastrogiteacihosting