Notes

Why open ports do not prove an NFS server is alive, why a working mount on another host proves nothing, and how a failed automount quietly sends writes to local disk.

How I took a Raspberry Pi back to a fresh-install state without reflashing it: reconstruct what was added from /var/log/apt/history.log, unmount NFS before touching the mountpoint, and reboot to clear leftover Docker networking.

Guard rails that stop a routine static-site deploy from turning into an account wipe: sentinel files, exclusion lists and deletion ceilings.

System services run without HOME unless the unit sets User=, DynamicUser=, PAMName= or SetLoginEnvironment=. Go tools that look up the home directory then fail instantly, and a health check that throws its output away turns that into an abort with no reason attached.

Relaying through a cPanel mail server with strict TLS fails with "hostname mismatch" even though the certificate is valid. Postfix does not send SNI by default; smtp_tls_servername = nexthop fixes it.

Without pipefail, a backup pipeline reports the exit status of gzip, not of pg_dump. Dump to a temporary file in custom format, check it with pg_restore -l, and only then rename it.

Why open() on a passed-through device node returns EPERM as root, how the cgroup v2 eBPF device filter causes it, and the short control test that tells you it is the cause.

Pod Security Admission enforces on Pods, not on workload templates. A privileged DaemonSet is created, reports healthy and never runs a pod, and a server-side dry run of it will not tell you.

mDNS and SSDP do not cross the pod network. A macvlan second interface through Multus gets around that, provided the node NIC, the CNI plugins and the attachment routes are all right. Each of them fails silently when it is not.

Longhorn always mounts an NFS backup target as nfs4, including in its fallback chain, so a NAS that only serves NFSv3 cannot be a backup target. How to check the server, and what enabling v4 on an older QNAP involves.

The default service range 10.96.0.0/12 reaches 10.111.255.255. If the node network sits inside it, Talos treats the node address as Kubernetes-only and leaves it out of the API certificate and etcd. The symptoms, how to see it with talosctl, and the smaller service range that fixes it.

From 1.14 Talos moves kubelet, labels, taints and the service range out of the v1alpha1 config into typed documents, and rejects the old fields next to them. A node on an older version rejects the new documents. What that does to patches, fixtures and upgrade order.

Talos labels control-plane nodes as excluded from external load balancers, and MetalLB speakers honour the label. On a cluster without workers that leaves no node to announce a layer 2 address. How the speaker log and servicel2status show it, and the patch that removes the label.

Without --iface, flannel records the address of whichever interface it found first as the node's VXLAN endpoint. Change that address and cross-node traffic dies while every node still looks healthy. The annotation to check and the restart that fixes it.

Talos binds kube-controller-manager, kube-scheduler and kube-proxy metrics to the loopback address, so kube-prometheus-stack shows nine dead targets. The machine-config patch, why kube-proxy needs a different treatment, and why talosctl upgrade-k8s is not the small operation it sounds like.

A macvlan parent interface left on DHCP installs a LAN default route at the same metric as the real one. Egress drifts onto the wrong network and replies to LAN clients take a different path than their requests. Why a /32 with no gateway is the fix, and the commands that show the problem.

Sixty evicted pods in an hour on a node with an 18 GiB disk. The kubelet's 15% imagefs threshold leaves about 1.4 GiB of headroom there, so one burst of image pulls evicts everything. How to read the numbers, clean up, and what pruning does and does not buy.

A Deployment from a Talos extraManifests entry was missing for at least a month and nothing noticed. How to tell that leftover objects came from Talos rather than GitOps, how to date a disappearance with kube-state-metrics, and what a missing kubelet serving-cert approver eventually costs.

The stock etcdHighFsyncDurations warning fired about 200 times in a week on a homelab. Why histogram bucket edges decide what a threshold means, how kube-prometheus-stack lets you change it, and why raising it hides a real disk problem.

I put an MQTT web client in front of Mosquitto and tried to keep the WebSocket port cluster-only. bind_interface does nothing for websockets listeners in 2.0.22, an interface name as host hangs the broker, and a pod IP rendered by an init container is what works.

Semaphore panicked on a bad Port value because Kubernetes injected SEMAPHORE_PORT=tcp://... for a Service with the same name. Turn off enableServiceLinks when the Service name matches the app, plus two migration gotchas from moving a config directory with a huge git checkout.

How to feed Homepage widget passwords from a SOPS Secret with HOMEPAGE_VAR placeholders, why annotation widgets cannot use them, and why rewriting a public repo's history does not undo a leak.

After a reboot Flux reported etcd timeouts and etcd warned about a slow disk. Talos has no per-process I/O view, but /proc/<pid>/io can be read through talosctl; here it pointed at a Longhorn snapshot purge.

FTLCONF_LOCAL_IPV4 does nothing in Pi-hole v6, and the "Primary IP addresses" panel is detection-only. The setting that controls what pi.hole resolves to is dns.reply.host.IPv4.

Some images start as root, chown their data directory and then switch user. runAsNonRoot breaks them. Dropping all capabilities and adding back CHOWN, SETUID and SETGID is the tightest setup that still starts.

After Flux applies a rotated Secret, the pod keeps the old value in its environment until it restarts. A 302 from the app does not prove the new password works.

Translating a Compose file to a Deployment, command: in Kubernetes silently drops the image ENTRYPOINT. Gotenberg lost its init wrapper and Paperless' exporter lost its environment. Check what the old container actually ran.

A Service without a selector plus a hand-written EndpointSlice points Traefik at any address, and the existing wildcard certificate and DNS keep working. The risky part is the cutover order, because Flux prunes less than you think.

MetalLB only hands out addresses from its own pool, so a pod that has to own a LAN address (a scanner target, say) needs a macvlan interface. The hard part is the handover: two hosts answering ARP for one address break both.

After each rolling reboot, Deployment pods stayed behind in phase Succeeded. The pod garbage collector only acts above 12,500 terminated pods, and lowering that would also eat CronJob history. A small CronJob that deletes only orphans owned by ReplicaSets, StatefulSets and DaemonSets does the job.

On Talos, the chart's etcd Service selects no pods, so it looks like dead weight. Turning the whole kubeEtcd block off also drops the etcd alert rules and the Grafana dashboard. Turn off only the Service and ServiceMonitor.

A 20 GiB Longhorn volume took 36.7 GiB per replica. Home Assistant wrote its daily 1.6 GB backup to the same volume, and seven daily snapshots each kept one. Sending backups off the volume fixed the growth.

Homepage asks every server in docker.yaml on each /api/services call. One host that times out delays the whole response by minutes, and the page renders no tiles. Time the API from inside the pod to find it.

kubectl apply --dry-run=server checks the HelmRelease CRD schema and stops there. Rendering the chart is the only cheap way to catch a mistake inside spec.values.

The differences that actually bit me when I moved a workflow from GitHub Actions to a self-hosted act_runner on Gitea.

flux bootstrap github offers token auth or SSH deploy keys. Both need a personal access token at bootstrap; they differ in what the cluster keeps afterwards, and that is what to choose on.

When the thing you want declared lives in an application database behind an API, a CronJob that writes it every few minutes can do the operator's job. What went wrong for me was in the shell, in the API verbs, and in triggering the job by hand under Flux.

If the Flux Kustomization that applies a *.sops.yaml Secret has no decryption block, Flux applies the encrypted text as the value and reports success. The app then fails with a baffling auth error. How to spot it without reading the secret.

Handing a kubectl-applied add-on or Secret to Flux should change nothing. Two read-only checks prove it: kubectl diff and a server-side dry run under Flux's own field manager. And why you must never print the diff of a Secret.

Production applies these fine because the CRDs already exist. On a cluster rebuilt from nothing, a Kustomization with a HelmRelease and objects of the kinds that chart installs fails its dry run. Monitors for a Prometheus operator that is not there fail the same way. Split it into two stages.

A Longhorn volume showed up as detached and unused because Helm never deletes PVCs in the namespace a release moved away from. How to spot the orphan by comparing volumes with running pods, and why the reclaim policy decides how much a delete takes with it.

Gitea Actions jobs sat queued because act_runner was registering as ephemeral. The image only checks that GITEA_RUNNER_EPHEMERAL is non-empty, so setting it to "0" enables it, and the pod looks like a crash loop.

An unquoted Discord or Telegram user ID in stringData becomes a YAML number. The apply fails, and a long ID has already lost precision by the time you read the error.

A Kustomization that keeps failing flips its Ready condition between False and Unknown on every reconcile, and lastTransitionTime moves with it. An alert for "not ready for ten minutes" built on that field almost never fires. Keep your own first-seen time.

Gitea pull mirrors created through the migrate API: the request body, why private repos store a GitHub token in Gitea, what breaks when that token rotates, and how to prove each mirror is complete.

Targeting one VM pulled an orphaned image download in through a local value and planned its destroy. A temporary edit keeps it out, and applying a saved, inspected plan beats an auto-approved one, because a saved plan goes stale after any other apply.

An old NAS with Python 3.5 is a raw-only Ansible target. Everything then goes through a busybox shell that discards background jobs, reports success regardless, and has to be verified by its side effects.

The ansible pip package already ships its collections. Installing them again into the project directory puts newer copies first on the collections path, silently replacing the versions you froze.

If one task sets check_mode itself, the command line flag no longer reaches it, and ansible-playbook --check -e apply=true mutates for real. Fold ansible_check_mode into the variable the task uses.

The API server rewrites stringData into data, so kubernetes.core.k8s sees a difference each time. Use data with b64encode. Also: omit does not work inside a nested definition, so merge optional keys with combine.

A regex written as '\\s' inside a >- block never matched, so every node in my upgrade role looked outdated. Backslashes are literal in folded and literal scalars; double them only in double-quoted strings.

Merging the same Talos context twice renames it to name-1 and makes it current; talosctl kubeconfig --merge issues a new client certificate on every call. How to add a cluster to your default configs from automation without surprises.

The dry run prints its summary and diff to stderr, no drift is the text No changes, and a regex that anchors at the start of the string makes a role re-apply every node every run. Plus how to find what forces a reboot, and why failed applies must never be logged.

ansible -m command -a with docker ps --format {{.Names}} fails, because Ansible runs the argument through Jinja before the shell ever sees it. Wrap the Go template in a raw block.

A leftover state file makes Molecule report "instances already prepared" and skip the wipe. Run molecule reset first. Also: a cold start can outlast a 15 minute health gate.

Encrypt one value with ansible-vault encrypt_string into its own small vars file, so the big encrypted vault stays untouched, and prove nothing changed by comparing hashes per host.

Before a NAS reboots, everything using it has to stop. The inventory is wrong about what that is, umount does not release the mount, and the restore has to replay what was actually running.

An NFS mount using the QNAP filesystem path instead of the export name resolved to a different share, and an image that runs chown -R at startup re-owned almost everything on it. How to mount correctly, how to verify by inode, and why to keep the chown output.

The wildcard record was correct at the registrar, the public resolvers answered it, and the router and Pi-hole still said NXDOMAIN. They had cached the failure from a lookup made before the record existed, for the zone's negative TTL. How to tell, and the flush order.

influx backup has no incremental mode, and the backup of a 5 GB database takes 18 minutes while Home Assistant keeps writing. Restore a second backup into a scratch bucket and copy only the gap across with Flux to().

Home Assistant's InfluxDB integration can live in a UI config entry instead of configuration.yaml. Before moving or retiring the database, find the real writers in its own log.

Working out whether a mini PC can take a second NVMe: lspci only shows drives that are installed, dmidecode lists firmware defaults, and a generic board may not even say its model.

MEGAcmd restarted every 14 seconds with "Unable to connect to service". The real reason was an fcntl lock on an NFSv3 mount that outlived the killed server. Giving the lock file a new inode fixed it.

An LVM thin pool on Proxmox was 85 % allocated while the data inside used a third of that. Longhorn's volume trim moved nothing. fstrim on each node's filesystem took the pool to 25 % in five minutes.

Shared hosting FTP certificates usually cover the server node, not ftp.<your-domain>. Read the certificate, connect by the name it covers, and never turn verification off on a session carrying a password.

Pi-hole v6 shows clients as bare IPs unless you give them a description. A short stdlib Python script reads names from the UniFi controller and writes them through the Pi-hole API, then runs on a schedule as a CronJob. The API details and the traps in both.

Configure Shiki with two themes and no default colour, choose between them in CSS with prefers-color-scheme, and use a Shiki transformer, not a rehype plugin, to add file names and a copy button. Includes the contrast catch with the standard GitHub themes.

Keep the instructions in AGENTS.md and make CLAUDE.md a single line that imports it. And grep your global instructions before you rename anything, because a renamed file they point at breaks every session without an error.