Notes

Why open ports do not prove an NFS server is alive, why a working mount on another host proves nothing, and how a failed automount quietly sends writes to local disk.

How I took a Raspberry Pi back to a fresh-install state without reflashing it: reconstruct what was added from /var/log/apt/history.log, unmount NFS before touching the mountpoint, and reboot to clear leftover Docker networking.

Guard rails that stop a routine static-site deploy from turning into an account wipe: sentinel files, exclusion lists and deletion ceilings.

System services run without HOME unless the unit sets User=, DynamicUser=, PAMName= or SetLoginEnvironment=. Go tools that look up the home directory then fail instantly, and a health check that throws its output away turns that into an abort with no reason attached.

Relaying through a cPanel mail server with strict TLS fails with "hostname mismatch" even though the certificate is valid. Postfix does not send SNI by default; smtp_tls_servername = nexthop fixes it.

Why open() on a passed-through device node returns EPERM as root, how the cgroup v2 eBPF device filter causes it, and the short control test that tells you it is the cause.

Pod Security Admission enforces on Pods, not on workload templates. A privileged DaemonSet is created, reports healthy and never runs a pod, and a server-side dry run of it will not tell you.

mDNS and SSDP do not cross the pod network. A macvlan second interface through Multus gets around that, provided the node NIC, the CNI plugins and the attachment routes are all right. Each of them fails silently when it is not.

Longhorn always mounts an NFS backup target as nfs4, including in its fallback chain, so a NAS that only serves NFSv3 cannot be a backup target. How to check the server, and what enabling v4 on an older QNAP involves.

The default service range 10.96.0.0/12 reaches 10.111.255.255. If the node network sits inside it, Talos treats the node address as Kubernetes-only and leaves it out of the API certificate and etcd. The symptoms, how to see it with talosctl, and the smaller service range that fixes it.

From 1.14 Talos moves kubelet, labels, taints and the service range out of the v1alpha1 config into typed documents, and rejects the old fields next to them. A node on an older version rejects the new documents. What that does to patches, fixtures and upgrade order.

Talos labels control-plane nodes as excluded from external load balancers, and MetalLB speakers honour the label. On a cluster without workers that leaves no node to announce a layer 2 address. How the speaker log and servicel2status show it, and the patch that removes the label.

Without --iface, flannel records the address of whichever interface it found first as the node's VXLAN endpoint. Change that address and cross-node traffic dies while every node still looks healthy. The annotation to check and the restart that fixes it.

Talos binds kube-controller-manager, kube-scheduler and kube-proxy metrics to the loopback address, so kube-prometheus-stack shows nine dead targets. The machine-config patch, why kube-proxy needs a different treatment, and why talosctl upgrade-k8s is not the small operation it sounds like.

A macvlan parent interface left on DHCP installs a LAN default route at the same metric as the real one. Egress drifts onto the wrong network and replies to LAN clients take a different path than their requests. Why a /32 with no gateway is the fix, and the commands that show the problem.

Sixty evicted pods in an hour on a node with an 18 GiB disk. The kubelet's 15% imagefs threshold leaves about 1.4 GiB of headroom there, so one burst of image pulls evicts everything. How to read the numbers, clean up, and what pruning does and does not buy.

A Deployment from a Talos extraManifests entry was missing for at least a month and nothing noticed. How to tell that leftover objects came from Talos rather than GitOps, how to date a disappearance with kube-state-metrics, and what a missing kubelet serving-cert approver eventually costs.

The stock etcdHighFsyncDurations warning fired about 200 times in a week on a homelab. Why histogram bucket edges decide what a threshold means, how kube-prometheus-stack lets you change it, and why raising it hides a real disk problem.

I put an MQTT web client in front of Mosquitto and tried to keep the WebSocket port cluster-only. bind_interface does nothing for websockets listeners in 2.0.22, an interface name as host hangs the broker, and a pod IP rendered by an init container is what works.

Semaphore panicked on a bad Port value because Kubernetes injected SEMAPHORE_PORT=tcp://... for a Service with the same name. Turn off enableServiceLinks when the Service name matches the app, plus two migration gotchas from moving a config directory with a huge git checkout.

How to feed Homepage widget passwords from a SOPS Secret with HOMEPAGE_VAR placeholders, why annotation widgets cannot use them, and why rewriting a public repo's history does not undo a leak.

kubectl apply --dry-run=server checks the HelmRelease CRD schema and stops there. Rendering the chart is the only cheap way to catch a mistake inside spec.values.

The differences that actually bit me when I moved a workflow from GitHub Actions to a self-hosted act_runner on Gitea.

flux bootstrap github offers token auth or SSH deploy keys. Both need a personal access token at bootstrap; they differ in what the cluster keeps afterwards, and that is what to choose on.

When the thing you want declared lives in an application database behind an API, a CronJob that writes it every few minutes can do the operator's job. What went wrong for me was in the shell, in the API verbs, and in triggering the job by hand under Flux.

If the Flux Kustomization that applies a *.sops.yaml Secret has no decryption block, Flux applies the encrypted text as the value and reports success. The app then fails with a baffling auth error. How to spot it without reading the secret.

Handing a kubectl-applied add-on or Secret to Flux should change nothing. Two read-only checks prove it: kubectl diff and a server-side dry run under Flux's own field manager. And why you must never print the diff of a Secret.

Production applies these fine because the CRDs already exist. On a cluster rebuilt from nothing, a Kustomization with a HelmRelease and objects of the kinds that chart installs fails its dry run. Monitors for a Prometheus operator that is not there fail the same way. Split it into two stages.

A Longhorn volume showed up as detached and unused because Helm never deletes PVCs in the namespace a release moved away from. How to spot the orphan by comparing volumes with running pods, and why the reclaim policy decides how much a delete takes with it.

Gitea Actions jobs sat queued because act_runner was registering as ephemeral. The image only checks that GITEA_RUNNER_EPHEMERAL is non-empty, so setting it to "0" enables it, and the pod looks like a crash loop.

Gitea pull mirrors created through the migrate API: the request body, why private repos store a GitHub token in Gitea, what breaks when that token rotates, and how to prove each mirror is complete.

Targeting one VM pulled an orphaned image download in through a local value and planned its destroy. A temporary edit keeps it out, and applying a saved, inspected plan beats an auto-approved one, because a saved plan goes stale after any other apply.

An old NAS with Python 3.5 is a raw-only Ansible target. Everything then goes through a busybox shell that discards background jobs, reports success regardless, and has to be verified by its side effects.

The ansible pip package already ships its collections. Installing them again into the project directory puts newer copies first on the collections path, silently replacing the versions you froze.

If one task sets check_mode itself, the command line flag no longer reaches it, and ansible-playbook --check -e apply=true mutates for real. Fold ansible_check_mode into the variable the task uses.

The API server rewrites stringData into data, so kubernetes.core.k8s sees a difference each time. Use data with b64encode. Also: omit does not work inside a nested definition, so merge optional keys with combine.

A regex written as '\\s' inside a >- block never matched, so every node in my upgrade role looked outdated. Backslashes are literal in folded and literal scalars; double them only in double-quoted strings.

Merging the same Talos context twice renames it to name-1 and makes it current; talosctl kubeconfig --merge issues a new client certificate on every call. How to add a cluster to your default configs from automation without surprises.

The dry run prints its summary and diff to stderr, no drift is the text No changes, and a regex that anchors at the start of the string makes a role re-apply every node every run. Plus how to find what forces a reboot, and why failed applies must never be logged.

Before a NAS reboots, everything using it has to stop. The inventory is wrong about what that is, umount does not release the mount, and the restore has to replay what was actually running.

An NFS mount using the QNAP filesystem path instead of the export name resolved to a different share, and an image that runs chown -R at startup re-owned almost everything on it. How to mount correctly, how to verify by inode, and why to keep the chown output.

The wildcard record was correct at the registrar, the public resolvers answered it, and the router and Pi-hole still said NXDOMAIN. They had cached the failure from a lookup made before the record existed, for the zone's negative TTL. How to tell, and the flush order.

Shared hosting FTP certificates usually cover the server node, not ftp.<your-domain>. Read the certificate, connect by the name it covers, and never turn verification off on a session carrying a password.

Pi-hole v6 shows clients as bare IPs unless you give them a description. A short stdlib Python script reads names from the UniFi controller and writes them through the Pi-hole API, then runs on a schedule as a CronJob. The API details and the traps in both.

Configure Shiki with two themes and no default colour, choose between them in CSS with prefers-color-scheme, and use a Shiki transformer, not a rehype plugin, to add file names and a copy button. Includes the contrast catch with the standard GitHub themes.