Flannel remembers the interface it started on

Without --iface, flannel records the address of whichever interface it found first as the node's VXLAN endpoint. Change that address and cross-node traffic dies while every node still looks healthy. The annotation to check and the restart that fixes it.

I removed the DHCP lease from a node’s second network interface. Within minutes, Flux couldn’t fetch from GitHub: lookup github.com on <cluster DNS>:53: i/o timeout. CoreDNS pods were healthy and logging. Nodes were Ready. A throwaway pod asking the cluster DNS timed out, while the same pod asking the router or the Pi-hole directly got answers.

The pattern is the clue. DNS to a CoreDNS replica on another node fails, to the same node or to something outside the cluster works. That’s pod-to-pod traffic between nodes breaking, and that’s the CNI’s job.

What flannel recorded

Flannel started without an --iface argument, so it chose an interface by itself and wrote that interface’s address into a node annotation:

kubectl get nodes -o custom-columns='NODE:.metadata.name,IP:.metadata.annotations.flannel\.alpha\.coreos\.com/public-ip'

Mine showed addresses that no longer existed. The VXLAN tunnel between nodes is built from those values, so every packet crossing a node boundary went to a dead endpoint. Flannel doesn’t notice that an address went away; it recorded it once.

Fix

Restart the DaemonSet so each pod detects the interface again:

kubectl -n kube-system rollout restart ds/kube-flannel

The annotations updated, cluster DNS answered on every try, and Flux caught up on its own.

Pinning it so it can’t happen again

The permanent version is to give flannel the interface by name: --iface=<the cluster-facing NIC>. On Talos that’s cluster.network.cni.flannel.extraArgs in the machine config. The catch is that flannel is a bootstrap manifest. Changing the machine config doesn’t touch the DaemonSet that’s already running. The new argument reaches it through talosctl upgrade-k8s, or by editing the DaemonSet yourself. Check the live args afterwards rather than trusting the config.

The rule I took from it: after any change to a node’s addresses or default route, look at that annotation on every node before you start debugging DNS. The change that triggered this was a second NIC picking up a default route it shouldn’t have had.