Why Prometheus cannot scrape the control plane on Talos, and the patch that fixes it

Talos binds kube-controller-manager, kube-scheduler and kube-proxy metrics to the loopback address, so kube-prometheus-stack shows nine dead targets. The machine-config patch, why kube-proxy needs a different treatment, and why talosctl upgrade-k8s is not the small operation it sounds like.

Three alerts had been firing for as long as I could remember: the controller manager, the scheduler and kube-proxy were all “unreachable” on every node. I had half decided they were noise, which is a bad habit for alerts. When I finally looked at the Prometheus targets page, all nine targets (three components on three nodes) were down with connection refused, while etcd, which I had patched earlier, was fine.

What is actually wrong

Nothing is broken. Talos starts kube-controller-manager and kube-scheduler with a loopback-only bind address, and kube-proxy exposes its metrics on the loopback address too. Prometheus scrapes the node IP, so it never reaches them. kube-prometheus-stack ships scrape targets for all three, which is why every Talos cluster that installs it gets this alert set for free.

The patch

For the controller manager and the scheduler it is a plain machine-config change:

cluster:
  controllerManager:
    extraArgs:
      bind-address: 0.0.0.0
  scheduler:
    extraArgs:
      bind-address: 0.0.0.0

Apply it per node with talosctl patch machineconfig --mode=no-reboot. No reboot is needed; Talos restarts those static pods itself, and they were back with the new flag within 25 to 50 seconds. I went one node at a time and checked etcd health before the next one.

The secure ports still require authentication for /metrics, and the Prometheus service account has that, so opening the bind address does not make the metrics public. kube-proxy is different: its metrics port speaks plain HTTP with no auth, and 0.0.0.0 means it is now reachable from every network the node sits on. On a flat home LAN I accepted that. On anything shared I would think again.

kube-proxy is not part of the machine config

This is the part that cost me time. On Talos, kube-proxy is a bootstrap manifest, a DaemonSet that Talos creates once. Changing the machine config does nothing to it. The documented way to sync it is talosctl upgrade-k8s --to <current version>, and that sounds like a no-op when you give it the version you already run.

It isn’t. It re-applies every bootstrap manifest and every URL listed under cluster.extraManifests. When I ran it with --dry-run it wanted to touch metrics-server, and to create a Deployment that had gone missing months earlier (see Talos applies extraManifests once). That is a lot of change to accept for one flag. If an extraManifests URL points at latest or main, the run also pulls whatever is there now.

So I patched the DaemonSet directly instead, adding --metrics-bind-address=0.0.0.0:10249 to the container command with a JSON patch. The argument order matters: Talos keeps the flags alphabetical, and once mine sat in the same place the dry-run reported the DaemonSet as unchanged. From then on the next upgrade-k8s has nothing to fix there.

What I would do again

Change the live nodes with talosctl patch machineconfig, and put the same patch into whatever generates your configs so a rebuild gets it. Running apply-config from a role wipes per-node patches, and I had already been bitten by that once. Afterwards Prometheus showed nine of nine targets up, and the three permanent alerts went away.

If you do not want these metrics at all, the other honest option is to disable kubeControllerManager, kubeScheduler and kubeProxy in the chart values. That removes the targets and the rules together. What you should not do is leave the alerts firing and learn to ignore them.