MetalLB allocates an address and announces nothing when every node is a control plane

Talos labels control-plane nodes as excluded from external load balancers, and MetalLB speakers honour the label. On a cluster without workers that leaves no node to announce a layer 2 address. How the speaker log and servicel2status show it, and the patch that removes the label.

The service had its address and the controller said so: IPAllocated, pool matched, address pinned by annotation. Nothing answered on it. No ARP reply, no connection, and the address wasn’t reachable from the LAN at all.

Pinging it isn’t a test, by the way. A load-balancer address isn’t configured on any interface, so a node won’t answer ICMP for it. Test the service port.

Where the answer was

The speaker pods run MetalLB’s layer 2 announcements, and they log why they skip a service when debug logging is on:

kubectl -n metallb-system logs <speaker-pod> | grep -v 'announced interface to exclude'

Every speaker said the same thing: speaker’s node has labeled node.kubernetes.io/exclude-from-external-load-balancers, followed by “no available nodes”. The other pointer is the status object MetalLB creates when a node does announce:

kubectl get servicel2status -A      # empty here: nobody had claimed the address

The cause is the label. Talos puts it on control-plane nodes, and the MetalLB speaker takes it seriously. My test cluster has three control planes and no workers, and schedules workloads on the control planes, so every node carried the label and none could announce.

Fix

Remove the label in the machine config. Which way depends on the Talos version:

machine:
  nodeLabels:
    node.kubernetes.io/exclude-from-external-load-balancers:
      $patch: delete

apiVersion: v1alpha1
kind: KubeNodeConfig
labels:
  node.kubernetes.io/exclude-from-external-load-balancers:
    $patch: delete

Applied live, no reboot. Talos removed the label from the existing Node objects as well, and a speaker took over the address. The two forms don’t mix: the first is rejected next to the typed document, so a cluster that moves from 1.13 to 1.14 has to rewrite the patch.