A service range that covers your node network hides the node address from Talos

The default service range 10.96.0.0/12 reaches 10.111.255.255. If the node network sits inside it, Talos treats the node address as Kubernetes-only and leaves it out of the API certificate and etcd. The symptoms, how to see it with talosctl, and the smaller service range that fixes it.

My first test nodes refused every connection to their own address. The error was a TLS one: the API certificate was valid for loopback and for the address of the node’s second NIC, but not for the address I was connecting to. A few minutes later etcd sat at “Waiting for etcd spec” and never started.

The cause was a range I hadn’t chosen. The default service range is 10.96.0.0/12, which runs up to 10.111.255.255. My node network is a /24 somewhere in that block. Talos sees a node address that falls inside the service range and decides it must be a Kubernetes-internal address. It puts it in the “only-k8s” set, and that set is left out of two places that matter: the list of names in the API server certificate, and the addresses etcd advertises to its peers.

Seeing it

talosctl -n <node> get nodeaddress

The node’s address shows up in the only-k8s list, not in the default addresses. That one output explained both symptoms at once.

Fix

Give the cluster a service range that doesn’t overlap the node network. Production here uses a /23:

apiVersion: v1alpha1
kind: KubeNetworkConfig
serviceSubnets:
  - 10.96.0.0/23

On older Talos versions the same setting is cluster.network.serviceSubnets.

I also set cluster.etcd.advertisedSubnets to the node network. With a second NIC on the node, etcd otherwise advertises whichever address it likes best, and that was the next problem I hit.

The addresses in the block above are the usual defaults, not something of mine.