Build logs, notes and project write-ups on infrastructure automation, Kubernetes, GitOps and the homelab they run on.https://cs-ol.de/https://cs-ol.de/blog/20261019-k8s-homelab-progress/https://cs-ol.de/blog/20261019-k8s-homelab-progress/Progress on the homelab cluster over the last ten days: InfluxDB and the document archive moved in, the old Compose host runs nothing, the node disks grew from 20 to 50 GiB the hard way, and a few things moved out again.Mon, 19 Oct 2026 00:00:00 GMThttps://cs-ol.de/blog/a-new-mega-sync-starts-without-history/https://cs-ol.de/blog/a-new-mega-sync-starts-without-history/Moving a MEGAcmd sync to a new host turned every file that had changed since its last upload into a conflict. On top of that, half the account quota was old versions and the rubbish bin, the package installs its own apt source, and syncs are two-way only.Sun, 11 Oct 2026 00:00:00 GMTbackupshttps://cs-ol.de/blog/flux-suspend-does-not-hold-a-child-kustomization/https://cs-ol.de/blog/flux-suspend-does-not-hold-a-child-kustomization/A correction to my NAS maintenance notes: when a parent Kustomization applies the child, the next sync clears a manual suspend. Annotating the one object with reconcile=disabled holds, but it also stops Flux from pruning it.Sat, 10 Oct 2026 00:00:00 GMThttps://cs-ol.de/blog/growing-the-node-disks-fixed-eviction-and-hurt-everything-else/https://cs-ol.de/blog/growing-the-node-disks-fixed-eviction-and-hurt-everything-else/The kubelet evicted pods because the node disks were tiny, and the cure was obviously bigger disks. Getting there took a trim of the thin pool, three reboots, and a night in which Home Assistant was down on and off for ninety minutes and etcd timed out because of one shared NVMe.Sat, 10 Oct 2026 00:00:00 GMTlonghornproxmoxetcdstoragehttps://cs-ol.de/blog/20261008-k8s-homelab-progress/https://cs-ol.de/blog/20261008-k8s-homelab-progress/Taking the Talos collection from a green test cluster to the live one: the drift a read-only check showed, three bugs only production could find, and a feature I moved and then deleted because Flux already did it.Thu, 08 Oct 2026 00:00:00 GMTtestinghttps://cs-ol.de/blog/the-node-that-came-back-on-the-wrong-address/https://cs-ol.de/blog/the-node-that-came-back-on-the-wrong-address/Applying a new machine config to a three-node cluster rebooted each node in turn. One came back on a different IP, because its DHCP reservation had been wrong for months and the node had never rebooted to notice.Wed, 07 Oct 2026 00:00:00 GMTproxmoxhttps://cs-ol.de/blog/testing-a-talos-collection-without-a-cluster/https://cs-ol.de/blog/testing-a-talos-collection-without-a-cluster/I wanted real-cluster tests for an Ansible collection that drives Talos. On my machine neither Docker nor QEMU would give me a node in maintenance mode, so I tested the rendering instead: fixtures, talosctl validate, determinism and negative cases. What it caught, and the five CI runs it took to go green.Sat, 03 Oct 2026 00:00:00 GMTtestingcigiteahttps://cs-ol.de/blog/pihole-answered-every-query-and-logged-none/https://cs-ol.de/blog/pihole-answered-every-query-and-logged-none/DNS lookups to the Pi-hole address succeeded, but its query log stayed empty. The answers came from the router, whose ad-block feature redirects every port-53 packet on the LAN to its own resolver. How I found out, and the one-line test that would have saved me the afternoon.Fri, 02 Oct 2026 00:00:00 GMTdnspi-holewslmetallbhttps://cs-ol.de/blog/one-corrupt-wal-segment-ate-prometheus-history/https://cs-ol.de/blog/one-corrupt-wal-segment-ate-prometheus-history/A Prometheus volume looked almost full while retentionSize was set. The real story: a single corrupt WAL segment had stopped every checkpoint for weeks, the WAL grew to 48.7 GiB, and the size cap spent the difference by deleting old blocks.Sat, 26 Sep 2026 00:00:00 GMTlonghornstoragemonitoringhttps://cs-ol.de/blog/after-the-nvme-what-the-fixes-bought/https://cs-ol.de/blog/after-the-nvme-what-the-fixes-bought/A follow-up on the cluster whose etcd shared one consumer drive with every storage replica. Flush latency fell from 23 ms to 3 ms, a filtered copy shrank a 9.9 GB SQLite database to 3.8 GB in ten minutes, and turning on discard cost more in replica rebuilds than in reboots.Fri, 25 Sep 2026 00:00:00 GMTetcdstoragelonghornproxmoxsqlitehttps://cs-ol.de/blog/three-etcd-members-one-consumer-nvme/https://cs-ol.de/blog/three-etcd-members-one-consumer-nvme/A scale-down that never happened led me to leader elections lost on every node, and from there to a single drive holding every etcd member and every storage replica. Three replicas on one disk mostly just triple the writes.Wed, 23 Sep 2026 00:00:00 GMTetcdstoragelonghornproxmoxhttps://cs-ol.de/blog/active-passive-failover-one-vip/https://cs-ol.de/blog/active-passive-failover-one-vip/Kubernetes has no notion of an active/passive Service. MetalLB in L2 mode with externalTrafficPolicy: Local is one, and I had been running it with a single replica the whole time.Mon, 21 Sep 2026 00:00:00 GMTmetallbdnshigh-availabilityhttps://cs-ol.de/blog/backup-job-that-backs-up-nothing/https://cs-ol.de/blog/backup-job-that-backs-up-nothing/The backup command I was asked to schedule did nothing on the current version: it printed usage text, exited 0 and wrote no file. So the job now checks the file it produces and refuses to upload anything that is not a real archive.Mon, 21 Sep 2026 00:00:00 GMTbackupspi-holesshhttps://cs-ol.de/blog/reranking-made-search-worse/https://cs-ol.de/blog/reranking-made-search-worse/I added local semantic search to a small knowledge base, and the default LLM reranker took 33 seconds to put the wrong document first. Every cheaper path got it right, which I only found out by asking questions I already knew the answers to.Sun, 13 Sep 2026 00:00:00 GMTsearchllmwslhttps://cs-ol.de/blog/static-site-on-shared-hosting/https://cs-ol.de/blog/static-site-on-shared-hosting/Retiring an unpatched WordPress on PHP 5.6 by replacing it with an Astro build pushed from a self-hosted Gitea runner to the same shared hosting.Sat, 12 Sep 2026 00:00:00 GMTastrogiteacihosting