Talos applies extraManifests once, and never checks on them again

A Deployment from a Talos extraManifests entry was missing for at least a month and nothing noticed. How to tell that leftover objects came from Talos rather than GitOps, how to date a disappearance with kube-state-metrics, and what a missing kubelet serving-cert approver eventually costs.

While reading the output of a dry run I noticed a Deployment that did not exist. The namespace, service account, cluster roles and binding for a kubelet serving-cert approver were all there, but no Deployment and no pod. I assumed GitOps had left it half removed, since I had tried and dropped a Flux install of it earlier.

It wasn’t GitOps

Git history did show a HelmRelease being added and removed on the same day, and the Helm repository URL it used returns 404, so that install could never have worked. The objects that were actually there had something else in common: a creation timestamp identical to metrics-server’s, to the second, and a last-applied-configuration annotation.

That is what Talos leaves behind. talosctl get mc listed the approver’s upstream install manifest and metrics-server under cluster.extraManifests. So the rule I took away: leftover objects with that annotation, absent from every Flux inventory, are Talos’s. Look in the machine config before digging through Git.

Talos does not reconcile them

Talos fetches those URLs and applies them once, when the node bootstraps or when you run talosctl upgrade-k8s. After that the objects are yours. If one gets deleted, or was never created properly, nothing brings it back until the next upgrade run re-applies everything. Mine had been missing for at least the whole Prometheus retention of 30 days, and the only reason I found out was luck.

You can date a disappearance with kube-state-metrics. kube_deployment_created and kube_pod_info for the namespace were empty for the whole window, while kube_service_info was present throughout. It only reaches back as far as your retention, but events and audit logs were long gone, so it was the only evidence left.

Why a missing approver matters

With rotate-server-certificates: true, kubelets request a new serving certificate as the old one ages, and something has to approve the request. That something is the approver. Without it, the kubelet keeps using its current certificate until it expires, which for me is July 2027. After that, kubectl logs, kubectl exec and metrics-server scraping fail, and the error does not point anywhere near the approver.

The serving certificates on my nodes were issued thirteen minutes before the extraManifests apply, so the approver had probably never done any work; I had approved those requests by hand. It was a time bomb with a very long fuse.

Two small side findings

Talos rejects JSON6902 patches on multi-document machine configs. To replace a list, pass two strategic-merge patches in one call: the first with extraManifests: {$patch: delete}, the second setting the new list. They apply in order and --mode=no-reboot still works.

Strategic merge also appends to lists. My role re-applied its patch to the same config file on every run, and the locally kept config now held twenty copies of each manifest URL. Nothing breaks, but generate the base config fresh, or the list keeps growing.

If you want the approver, the durable fix is to manage it in GitOps from a source that exists, for example the upstream manifest pinned to a release tag as a Kustomize remote resource, rather than a moving main URL.